Automated application of drift correction to sample studied under electron microscope

ABSTRACT

Control system configured for sample tracking in an electron microscope environment registers a movement associated with a region of interest located within an active area of a sample under observation with an electron microscope. The registered movement includes at least one directional constituent. The region of interest is positioned within a field of view of the electron microscope. The control system directs an adjustment of the electron microscope control component to one or more of dynamically center and dynamically focus the view through the electron microscope of the region of interest. The adjustment comprises one or more of a magnitude element and a direction element.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/951,297 filed on Nov. 18, 2020, entitled “AUTOMATED APPLICATION OFDRIFT CORRECTION TO SAMPLE STUDIED UNDER ELECTRON MICROSCOPE”, which isa continuation of International Patent Application No. PCT/US2020/045937filed on Aug. 12, 2020, entitled “AUTOMATED APPLICATION OF DRIFTCORRECTION TO SAMPLE STUDIED UNDER ELECTRON MICROSCOPE”, which claimspriority to U.S. Provisional Patent Application No. 62/888,309 filed onAug. 16, 2019, entitled “AUTOMATED DRIFT CORRECTION TO SAMPLE BEINGSTUDIED UNDER ELECTRON MICROSCOP”, the contents of all which is herebyincorporated by reference in their entireties.

TECHNICAL FIELD

The present disclosure relates to the field of electron microscopy, andparticularly to a system for automated tracking of, and correcting for,drift occurring within a sample being studied under an electronmicroscope.

BACKGROUND

Camera and detector software suites presently available on electronmicroscopes typically correct for small movements by digitally shiftinga limited field of view across the full field area available to thecamera or detector. In most traditional studies done with an electronmicroscope, the sample is at room temperature with plenty of time tosettle into thermal equilibrium. Measuring any number of microscopeparameters, such as dose rate, energy loss or X-ray counts, for a givencoordinate is straight forward on a system that is not moving.Accordingly, shifting the field of view to correct for movementsoccurring in a region of interest of the sample under observation canfacilitate sharper images of a region of interest. Movements occurringin a region of interest of the sample under observation are typicallysmall and can often be at a rate that is degrees of magnitude less thanone nanometer per minute.

“In-situ” or “operando” studies involve applying or enabling dynamicchanges to a sample, for example, by undertaking actions such asmechanically altering, electrically probing, heating, cooling, andimaging the sample in a gas or a fluidic environment. It may beadvantageous for the microscopist to track a region of interest withinthe sample as it undergoes various changes over time. Measurementsrelated to various parameters associated with the sample under studywould need to be registered in order to comprehensively track thechanges in various parameter that occur as the sample moves. This isbecause the tracked changes cannot be tied back to the originalcoordinates without carefully considering the history as to how andwhere a given feature has moved during the course of the experiment.Unfortunately, the magnitude of sample movement can be out of the rangefor common cameras and detectors to digitally shift the field of view inan adequate fashion.

Accordingly, opportunities exist for providing a novel approach forautomating feature tracking and drift correction in an electronmicroscope when needed.

SUMMARY

This summary is provided to introduce in a simplified form concepts thatare further described in the following detailed descriptions. Thissummary is not intended to identify key features or essential featuresof the claimed subject matter, nor is it to be construed as limiting thescope of the claimed subject matter.

Disclosed herein is a control system configured for sample tracking forsample tracking in an electron microscope environment. The controlsystem comprises a memory, a processor, and a microscope controlcomponent. The control system is configured to register a movementassociated with a region of interest located within an active area of asample under observation with an electron microscope. The registeredmovement includes at least one directional constituent. The region ofinterest is positioned within a field of view of the electronmicroscope. The control system is further configured to direct anadjustment of the microscope control component to one or more of:dynamically center a view through the electron microscope of the regionof interest, and dynamically focus the view through the electronmicroscope of the region of interest. The adjustment comprises amagnitude element and/or a direction element. According to one or moreembodiments, the control system is further configured to apply anin-situ stimulus to the region of interest.

Further, disclosed herein is a control system configured to registermovement associated with a region of interest located within an activearea of a sample under observation with an electron microscope. Theregistered movement includes at least one directional constituent. Theregion of interest is positioned within a field of view of an electronmicroscope. The registered movement including at least one of an Xtranslation, Y translation, Z translation, alpha-tilt and a beta-tilt.The control system is further configured to direct an adjustment of anelectron microscope control component to one or more of dynamicallycenter a view through the electron microscope of the region of interest,and dynamically focus the view through the electron microscope of theregion of interest. The adjustment comprises one or more of a magnitudeelement, and a direction element.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing, as well as the following Detailed Description ofpreferred embodiments, is better understood when read in conjunctionwith the appended drawings. For the purposes of illustration, there isshown in the drawings exemplary embodiments; however, the presentlydisclosed subject matter is not limited to the specific methods andinstrumentalities disclosed.

The embodiments illustrated, described, and discussed herein areillustrative of the present invention. As these embodiments of thepresent invention are described with reference to illustrations, variousmodifications, or adaptations of the methods and or specific structuresdescribed may become apparent to those skilled in the art. It will beappreciated that modifications and variations are covered by the aboveteachings and within the scope of the appended claims without departingfrom the spirit and intended scope thereof. All such modifications,adaptations, or variations that rely upon the teachings of the presentinvention, and through which these teachings have advanced the art, areconsidered to be within the spirit and scope of the present invention.Hence, these descriptions and drawings should not be considered in alimiting sense, as it is understood that the present invention is in noway limited to only the embodiments illustrated.

FIGS. 1A and 1B are schematic representation of a control systemconfigured for sample tracking and drift correction in an electronmicroscope environment, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 2 is a schematic representation illustrating details of a reactivedrift correction process by the control system, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 3A and 3B are schematic representations illustrating an on-the-flylearning by the control system of unique x, y and z axes movements of anE-chip and a holder in combination of predictive behavior of where thedrift is expected to occur, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 4 is a schematic representation illustrating a module of thecontrol system that tracks pixel shifts over time to build driftvelocity and acceleration vectors, according to one or more embodimentsof the presently disclosed subject matter.

FIG. 5 is a graphical representation of a module that forms part of thecontrol system that is configured to allow a user to select a region ofinterest by drawing and then command the electron microscope to move andcenter the ROI in the field of view, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 6 is a graphical representation of a module that forms part of thecontrol system having a pre-drawn ROI that is configured to allow a userto command a new center position, whereby the sample or beam is moved bythe control system, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 7 is a graphical representation of a module that forms part of thecontrol system that is configured to support multiple ROI on a singleset of consecutive images, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 8 is a flow chart wherein a module that forms part of the controlsystem that uses drift vectors, background drift and/or a referencetemplate to determine when a movement occurring within a sample,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 9 is a flowchart illustration of a module that forms part of thecontrol system that is configured to trigger to camera, detector,microscope, or in-situ, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 10A and 10B are a flowchart illustrating a module that forms partof the control system that is configured to use a hierarchal control ofpositioners, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 11 is a graphical illustration of a module that forms part of thecontrol system that is configured to apply a digital correction on topof a physical correction and saving consecutive images as movies,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 12A and 12B are a flow chart illustrating a module that forms partof the control system that is configured to run an autofocus or refocusroutine to find the ideal focus, according to one or more embodiments ofthe presently disclosed subject matter.

FIG. 13 is a flow chart illustrating a focus scoring sweep, according toone or more embodiments of the presently disclosed subject matter.

FIG. 14 is a graphical representation of a visual focus control tool forelectron microscopes built from a normalized focus score versuscalculated ideal with user set refocus handles, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 15A and 15B are a graphical illustration of a module that formspart of the control system that is configured to combine positioner,lens and holder calibrations with actual behavior to improve directionand magnitude of commanded movements, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 16A-16B, FIG. 17A-B and FIG. 18A-B are flowcharts related to amodule that forms part of the control system that is configured tomonitor x-axis, y-axis and z-axis positions, alpha/beta tilt, and imagerefresh rate to flag any user interruptions, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 19 is a graphical illustration of a module that forms part of thecontrol system that is configured to trigger new behavior on the in-situcontrol, microscope, camera or detector from interruptions detected onthe microscope, according to one or more embodiments of the presentlydisclosed subject matter.

FIGS. 20A and 20B are a graphical illustration of a module that formspart of the control system that is configured to take user interruptionson the microscope and improves on expected models or processes,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 21 is a graphical illustration of a module that forms part of thecontrol system that is configured to provide automatic attenuation ofin-situ control inputs such as ramp rate to prevent the loss of theprimary ROI, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 22 is a flowchart of a module that forms part of the control systemthat is configured to calculate a maximum ramp rate of the stimulus fromthe active field of view relative to ROI size, positioner timing, imageupdate rate and expected drift rate, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 23 is a flowchart of a module that forms part of the control systemthat is configured to help a user set the magnification, active detectorsize, pixel resolution, binning, dwell rate and/or exposure time toachieve specific thermal ramp rates, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 24 is a schematic graphical representation of a module that formspart of the control system that is configured to allow a user toprioritize one or more camera/detector options, microscope setup, andin-situ stimulus to ensure a stable image within the capabilities ofdrift correction, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 25 is a schematic representation of a module that forms part of thecontrol system that is configured to apply drift vectors to predict thelocation of secondary or many other imaging sites and allowing users toeasily toggle between sites, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 26 is a schematic graphical representation of an indicator thatforms part of the control system that is configured to normalize driftrate and alert the user of when movement is slow enough for ahigh-resolution acquisition, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 27 is a diagrammatic representation of a module that forms part ofthe control system that is configured to enable a user or other softwaremodules to set triggers to the in-situ function based from imageanalysis, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 28 is a diagrammatic representation of a module that forms part ofthe control system that is configured to enable a user or anothersoftware module to set triggers to the electron microscope, camera ordetector, based from in-situ stimulus readings, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 29 is a diagrammatic representation of interface that form part ofthe control system that is configured to help researchers buildexperiments and make custom triggers, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 30 is a schematic representation of a module that forms part of thecontrol system that is configured to track a total dose and dose rate ofa specific sample site to help a user quantify beam damage of a site fora specific feature, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 31 and FIG. 32 are schematic graphical representations of avisualizer module that forms part of the control system that isconfigured to help a user compare beam effects for a single site atspecific times or for specific in-situ stimulus conditions, according toone or more embodiments of the presently disclosed subject matter.

FIG. 33 is a schematic graphical representation of an automatic reportgenerator module that forms part of the control system that isconfigured to compare sample sites as a function of time, according toone or more embodiments of the presently disclosed subject matter.

FIG. 34 is a schematic graphical representation of an automatic reportgenerator module that forms part of the control system that comparessample sites for a given in-situ control, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 35 and FIG. 36 are schematic graphical representations of a modulethat can form part of the control system that is configured to limitdose, dose rate or other microscope parameters as well as in-situstimulus, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 37 is a diagrammatic representation of an example for how multiplesample sites can be tracked across an entire imageable area for quicknavigation through UI or triggers, according to one or more embodimentsof the presently disclosed subject matter.

FIG. 38 is an illustrative representation of an example of one or moreregions of interest identified on a live image feed with key functionsto keep a sample stable in X, Y and Z aces included along with some keymetadata describing the image, according to one or more embodiments ofthe presently disclosed subject matter.

FIG. 39 is a schematic graphical representation of a basic communicationarchitecture for a software module that forms part of the controlsystem, according to one or more embodiments of the presently disclosedsubject matter.

FIG. 40 is a schematic graphical representation of a filtering techniquethat reduces background noise of an image, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 41 is a schematic graphical representation of multiple regions ofinterest presented against total field of view, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 42 is a schematic graphical representation of an example of reportgenerated from multiple sites for a given time period or a given in-situstimulus, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 43 is a schematic graphical representation of the control system inthe form of a chart, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 44A and 44B, FIG. 45, FIG. 46, FIGS. 47A and 47B, FIG. 48, FIG.49, FIG. 50, FIG. 51, FIGS. 52A and 52B, FIG. 53, FIG. 54, FIGS. 55A and55B, FIG. 56, and FIG. 57 illustrate various portions of the controlsystem of FIG. 43.

FIG. 58 is a graphical representation of the first step in an automatedexperimental workflow, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 59 is a graphical representation of the second step in an automatedexperimental workflow, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 60 is a graphical representation of the third step in an automatedexperimental workflow, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 61 is a graphical representation of the fourth step in an automatedexperimental workflow, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 62 is a graphical representation of the fifth step in an automatedexperimental workflow, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 63 is a graphical representation of the sixth step in an automatedexperimental workflow, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 64 is a graphical representation of an alternative view of thesixth step in an automated experimental workflow, according to one ormore embodiments of the presently disclosed subject matter.

FIG. 65 is a graphical representation of an alternative view of thesixth step in an automated experimental workflow, according to one ormore embodiments of the presently disclosed subject matter.

FIGS. 66A and 66B are a schematic graphical representation showing howtagged regions at multiple sites can be tracked even if only one regionof interest is in the field of view, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 67 is a schematic graphical representation of an architecture wherea control software running on a control software CPU utilizes a singlemicroscope service on the microscope CPU, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 68A and 68B are a schematic graphical representation of anarchitecture where a control software running on a control software CPUutilizes both a microscope service on the microscope CPU and an imagingservice on the imaging CPU, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 69A, 69B, and 69C are a schematic graphical representation of amicroscope service class needed for microscope commands and imagingcommands, according to one or more embodiments of the presentlydisclosed subject matter.

FIGS. 70A and 70B are a schematic graphical representation of amicroscope profile, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 71A, 71B, and 71C are a variation of FIG. 70 wherein themicroscope profile is created from content and capabilities from animaging service and a microscope service rather than a single service,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 72A, 72B, and 72C are a schematic graphical representation of ahigh-level process to connect to the microscope and an imaging softwaremodule and transmit unique images with all relevant metadata to thecontrol software module, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 73A, 73B, and 73C are a schematic graphical representation of amore detailed image monitoring process that can be used to determineunique images from a continuous image feed and transmit the uniqueimages to the control software module, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 74A and 74B are a schematic graphical representation of a processused to connect to the required services, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 75A and 75B are a schematic graphical representation of a testconnection process, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 76A, 76B, and 76C are a schematic graphical representation of aprocess to calibrate for the X/Y rotational offset between a positionerand an imager, according to one or more embodiments of the presentlydisclosed subject matter.

FIGS. 77A and 77B are a schematic graphical representation of a processto handle multiple positioners capable of calibrating under specificimaging conditions, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 78A and 78B are a schematic graphical representation of a processto calibrate the required Z adjustment needed to correct for an imagequality score change under specific imaging conditions, according to oneor more embodiments of the presently disclosed subject matter.

FIGS. 79A, 79B, 79C, and 79D are a schematic graphical representation ofa process to run drift correction in X, Y and Z, according to one ormore embodiments of the presently disclosed subject matter.

FIGS. 80A and 80B are a schematic graphical representation of a processto start image acquisition remotely from a control software module,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 81A and 81B are a schematic graphical representation of a processto stop image acquisition remotely from a control software module,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 82A and 82B are a schematic graphical representation of a processto move a sample to a specific location in the field of view, accordingto one or more embodiments of the presently disclosed subject matter.

FIGS. 83A and 83B are a schematic graphical representation of a processto determine if the image has stabilized after a commanded move by themicroscope, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 84 is a graphical representation of key controls and indicatorsthat could enhance the drift correction experience in the controlsoftware module user interface, according to one or more embodiments ofthe presently disclosed subject matter.

FIG. 85 is a graphical representation of key controls that can enableusers to review the history of a session from the software module userinterface, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 86 is a graphical representation of a method by which users couldtag specific frames and time sequences with a description from thecontrol software module user interface, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 87 is a graphical representation of key settings that a user couldmanipulate to customize the active image buffer and session management,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 88 and FIG. 89 are graphical representations of how the controlsoftware module could be used to build a microscope profile, accordingto one or more embodiments of the presently disclosed subject matter.

FIG. 90 and FIG. 91 are graphical representations of how the controlsoftware module could manage calibrations specific to imaging conditionsand imagers, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 92 is a graphical representation of a user interface enabling usersto dictate specific types of in-situ experiments or workflows that maychange the behavior or options of the control software module, accordingto one or more embodiments of the presently disclosed subject matter.

FIGS. 93A and 93B are a graphical representation of a user interfaceenabling key workflow functions, according to one or more embodiments ofthe presently disclosed subject matter.

FIGS. 94A, 94B, 94C, and 94D are a graphical representation of a userinterface comprised of indicators and triggers that enhance thecorrection experience, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 95A, 95B, 95C, 95D, and 95E are a graphical representation of auser interface for a session review tool where users can view images andmetadata, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 96 is a graphical representation of user settings that can bemanipulated to customize the experience, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 97 is a graphical representation of a user interface where focusassist and focus assist calibrations can be enabled while viewing thelive image, according to one or more embodiments of the presentlydisclosed subject matter.

FIGS. 98A, 98B, and 98C are a graphical representation of how thecontrol software module or associated documentation could communicatethe relationship between image acquisition rate and field of view as afunction of acceptable drift rate, according to one or more embodimentsof the presently disclosed subject matter.

FIGS. 99A and 99B are a graphical representation of how a focusalgorithm can utilize the focus quality score in STEM mode to drivetoward an apex through adjustment of defocus, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 100A and 100B are a graphical representation of how a focusalgorithm can utilize the inverse of the focus quality score in TEM modeto drive toward an apex through adjustment of defocus, according to oneor more embodiments of the presently disclosed subject matter.

FIG. 101 is a graphical representation of the overall data flow for acontrol service interacting with various components of the system,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 102A and 102B are a graphical representation of a user interfaceof an in-situ heating software module, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 103A and 103B a graphical representation of a user interface wherethe control software module recommends ramp rates and communicatesautomated pauses/resumes and connection status within an in-situsoftware module and a control software module, according to one or moreembodiments of the presently disclosed subject matter.

FIGS. 104A, 104B, 104C, 104D, and 104E are a graphical representation ofa user interface where metadata from the in-situ system, microscope,imaging system and any other connected systems can be viewed andoverlaid onto the live display and session or image review tool,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 105A, 105B, and 105C are a graphical representation showing anexample of an existing in-situ software module suite with uniqueworkflows and reporting elements pushing data to another software modulethat synchronizes data; and, FIG. 105B details an example of a workflowin an existing in-situ software vs the reporting elements in thatsoftware module, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 106 is a graphical representation showing how the software modulesuite described in FIG. 105A could have workflows shared between thenative in-situ software module and an embedded element within thecontrol software module, according to one or more embodiments of thepresently disclosed subject matter.

FIGS. 107A, 107B, 107C, and 107D are a graphical representation showingan example of the user interface of an existing in-situ software module,according to one or more embodiments of the presently disclosed subjectmatter.

FIGS. 108A and 108B and FIG. 109 are graphical representations of userinterfaces used for an existing in-situ control software module,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 110 through FIG. 115 are graphical flow charts detailing a workflowwhere a control software module can help users effectively quantify,knowingly operate within and review the effects of cumulative dose ormaximum instantaneous dose rate on an experiment, according to one ormore embodiments of the presently disclosed subject matter.

DETAILED DESCRIPTION OF EMBODIMENTS

Below, the technical solutions in the examples of the present inventionare depicted clearly and comprehensively with reference to the figuresaccording to the examples of the present invention. Obviously, theexamples depicted here are merely some examples, but not all examples ofthe present invention. In general, the components in the examples of thepresent invention depicted and shown in the figures herein can bearranged and designed according to different configurations. Thus,detailed description of the examples of the present invention providedin the figures below are not intended to limit the scope of the presentinvention as claimed, but merely represent selected examples of thepresent invention. On the basis of the examples of the presentinvention, other examples that could be obtained by a person skilled inthe art without using inventive efforts will fall within the scope ofprotection of the present invention. The invention will now be describedwith reference to the Figures shown below.

Transmission electron microscopy (TEM) uses a beam of electronstransmitted through a specimen to form an image. Scanning transmissionelectron microscopy (STEM) combines the principles of transmissionelectron microscopy and scanning electron microscopy (SEM) and can beperformed on either type of instrument. While in TEM parallel electronbeams are focused perpendicular to the sample plane, in STEM the beam isfocused at a large angle and is converged into a focal point. Like TEM,STEM requires very thin samples and looks primarily at beam electronstransmitted through the sample. One of the principal advantages of STEMover TEM is in enabling the use of other of signals that cannot bespatially correlated in TEM, including secondary electrons, scatteredbeam electrons, characteristic X-rays, and electron energy loss.

As a microscopist readily understands, “in-situ” or “operando” studiesinvolve applying or enabling dynamic changes to the sample, for example,by undertaking actions such as mechanically altering, electricallyprobing, heating, cooling, and imaging the sample in gas or fluidicenvironment. Traditional in-situ systems, MEMS (microelectromechanicalsystems) sample supports, and modern electron microscope holders havehelped reduce the movement associated with “in-situ” or “operando”studies by minimizing and localizing the stimulus to the sample area,but even these systems present too much movement to correct for usingany automation that may be presently available in the marketplace.

Traditional in-situ systems include bulk heating or furnace heatingholders that are capable of heating larger samples without a MEMS samplesupport. Bulk heating or furnace heating holders are better suited forstudying some samples such as polished metals because the samplepreparation process is unique and the size of sample requires too muchenergy that cannot be provided by MEMS sample supports in acost-effective manner. The large amount of energy required to heat suchbulk heating or furnace heating holders creates a lot of drift of thesample being studied. Physically correcting this drift can enableimaging at a higher magnification and a more stable, usable experience.

For example, during a thermal heating experiment, changing thetemperature a few hundred degrees can move the sample a few hundrednanometers in the x, y plane and often introduce a change in height inthe z-axis as materials expand and contract during the course ofachieving thermal equilibrium. There are a lot of other sources of driftin the x, y and z axes stemming from the microscope positioner systems,holder positioner system, optics, gun, or environmental changes notrelated to in-situ.

Common techniques such as EDS (Energy Dispersive X-Ray Spectroscopy) andEELS (Electron Energy Loss Spectroscopy) require the sample to be stillfor enough time in order to acquire adequate data—often in the magnitudeof several minutes. It is difficult for a person to run these techniquesall at the same time if the person is also tracking the features bymanually moving the holder or electron beam. Physical corrections enableworkflows where fast acquisitions or scans can be used over longerperiods of time building a “live” map of elemental analysis. Since thesample is physically corrected, the same sample can be imaged quicklygenerating smaller signals—but when summed into a running average, itcan create detailed maps of the sample over a time frame, possibly eventhrough in-situ environmental changes.

The sample holder is typically moved using a mechanical stage or agoniometer. A user would have to track the sample by manually andcontinuously moving the sample holder or electron beam to keep a regionof interest centered since the illumination, cameras, and detectors arefixedly positioned. There are stage controls provided for finermovements of the stage (i.e., the flat platform) that supports thesample under observation. These stage controls include piezo variations,with the controlling of the stage usually accomplished by the operationof a joystick or trackball. However, coordinates and jogs are oftencommanded from software suites supplied with the microscope. It is notuncommon to require 2 people to carry out the experiments—one forcontrolling the stimulus to the sample and another for operating themicroscope to account for sample movement. Under existing systems,measurements of a single feature must be manually tracked; also, suchmeasurements are typically tied to x, y, and z coordinates rather thanto specific features themselves.

During imaging of a sample using electron microscopy, the electron beamis typically directed on the sample during the entire process of imagingthe sample including the steps of locating the sample, focusing on thesample, and recording the image. The electron beam can cause damage tothe sample itself, and this damage is proportional to the total dose andthe dose rate. The electron dose for a given area (e−/Å{circumflex over( )}2) is an important parameter and is calculated by multiplying thecurrent density in the probe (Å/m²) by the exposure time (s). The doserate is a measured as the electron dose applied as a function of time.Beam damage can physically change a sample as chemical bonds get broken.The type and degree of damage from the electron beam depends on thecharacteristics of the beam and the sample. Numerous studies haveinvestigated how electron beams damage samples. One example is by way ofknock-on damage, wherein incident electrons transfer kinetic energy tothe sample which can displace atoms or sputter them from the surface ofthe sample. Another example is by way of radiolysis or ionization due toinelastic scattering; this type of damage is common in insulatingsamples or liquids. A further example is by way of electrostaticcharging of materials that is caused by the electron beam, which canlead to positive surface potentials due to ejected secondary or augerelectrons. However, reducing dose arbitrarily to limit damage candegrade image resolution, especially for beam sensitive samples.Ideally, the goal is to operate the microscope at the highest dosepossible without causing beam damage for a given sample; however,determining and staying under this “safe” dose/dose rate limit ischallenging. While radiation damage cannot be eliminated, it can bemeasured and minimized Since the electron-beam-induced radiation damageis proportional to the electron dose and dose rate, measuring andcontrolling electron dose and dose rate is an ideal solution to controland limit damage to the specimen.

To better understand the impact of electron dose on a given specimen, itwould be beneficial to measure, display, and record the cumulative doseimparted as a function of position on a specimen over the course of animaging session. It would also be helpful to be able to set limits onelectron dose and dose rate as a function of area to control beam damageto the sample during imaging. Further, with the continuous analysis andcontrol of the microscope, camera, detector and in-situ stimulus, itwould be beneficial to provide event triggers that can automateexperiments wherein conditions of a sample are adjusted automatically bya control system.

Embodiments of the presently disclosed subject matter can advantageouslyoperate to correct drift occurring during in-situ studies. Driftoccurring during in-situ studies is only one example of drift that canbe corrected by embodiments of the presently disclosed subject matter.For example, embodiments disclosed herein can also advantageouslyoperate to counteract drift that can occur from mechanical settling froma sample holder, mechanical settling from a microscope positionersystem, thermal drift from environments not related to in-situ, thermaldrift imparted by the optics or gun, and similar other components, andelectrical drift imparted by the optics or gun, and similar othercomponents. embodiments disclosed herein can also advantageously operateto counteract drift such as a thermal drift or an electrical drift fromoptics adjustments. For example, factors such as changing accelerationvoltage of the gun, power changes in correctors, or power changes in therest of the optics can cause drift.

Embodiments disclosed herein can advantageously correct all kinds ofdrift encountered during observation made with an electron microscopethereby enabling higher magnifications and more stable imagingregardless of the source of drift. Indeed, at a high enoughmagnification level, any drift from any source can require physicalcorrections as well associated corrections to all the dependenttechnologies that are enabled. At a high enough magnification level,digital registration will be limited even on more standard types ofdrift after settling time. For example, in addition to in-situenvironmental changes and stimulus, drift can also be caused bymechanical settling from the holder or microscope positioner systems,thermal drift from environments not related to in-situ, thermal orelectrical drift imparted by the optics or gun, and similar othersources. Embodiments disclosed herein can advantageously operate tocounteract drift from any source.

Microscopy is challenging and in-situ microscopy adds additionalcomplexity making the barrier to entry large and the chance of successsmall. Workflows associated with microscopy study require expertise andmultiple resources working simultaneously. Often a team of two or threepeople are required to run an experiment: a TEM expert optimizing theimaging conditions and managing the re-centering and focusing throughthe experiment, an in-situ equipment expert controlling the stimulus,and an observer watching the sample and resulting data. Additionally, itis difficult to organize this data aligning the massive number of imagesand data generated in a session. Embodiments disclosed herein canadvantageously operate to reduce the learning curve associated within-situ microscopy by decreasing the level of expertise required to runan experiment, expanding the potential community of in-situ researchersand applications.

At least one embodiment of the presently disclosed subject matterincludes an electron microscope control system (alternately referred tohereinafter as “control system” or “system”). The control system asdisclosed herein can allow users to see every moment, putting theemphasis back on the sample and not the associated equipment. Thecontrol system can enable imaging at higher resolutions through anentire experiment and provide an undistracted viewing and capture offormerly unobservable moments. The control system can make the processof data analysis faster, easier, and more accurate. It can continuouslysynchronize data with relevant experiment conditions and let usersprioritize the most important parameters and controls the system tooptimize the others.

In various embodiments, the control system can include software modulesthat interact with the many systems in a TEM lab. The control system canbe embodied as a server that is networked to other systems including theTEM column, cameras, detectors, and in-situ systems. In one embodiment,the control system comprises software that can be run on hardware suchas a server operating at a client site. The control system can provide arobust software solution where modules address workflows linking the labdigitally. The control system can synchronize the physical sample withthe column/detectors for stable images; it can further synchronize allsystem data in the experiment for fast, accurate publishing; it can alsosynchronize the parameter control to enable experiment prioritysettings. The control system can allow for the sample to be stable withunderstood movement vectors and all systems networked to this TEM hub.The control system can allow for automation and system synchronizationthat works with the user during a TEM session. This way, the operator isstill in control, but can focus the operator's effort on the samplerather than managing all the associated equipment. The control systemcan address four key issues with today's electron microscopy and in-situEM workflows: (1) reduce the steep learning curve for electronmicroscopy, especially in-situ EM; (2) reveal “the missing moments”; (3)consolidate the experiment data that currently is distributed acrossdifferent systems; and (4) serve as a base platform to enable thedevelopment of advanced modules.

The control system can provide for tracking background drift helps inthe event of a changing sample, so the software prioritizes the userspecified region of interest against many different background templatessegmented from the total field of view. The software forming part ofvarious embodiments of the presently disclosed subject matter can usereference templates and drift vectors or background drift to determinewhen a sample is changing, such change including aspects such as phasetransformations and coalescing. A changing sample typically requires anew reference template and can be quantified to flag other events.

In addition to correcting for drift, and recording the amount ofmovement in the x, y and z axes over time, embodiments of the presentlydisclosed subject matter can also provide for recording athree-dimensional map of where the sample has traveled.

Embodiments of the presently disclosed subject matter can furtherprovide for displaying an interactive three-dimensional map on a GUI(graphical user display). In a liquid cell, for example, where samplemovement can be the result of a phenomenon under investigation, thecontrol system can provide for the drift correction vectors to bevisualized in a software tool that shows the three-dimensional path thesample took throughout the experiment. The control system can furtherprovide for such a 3D map could be visualized and rotated throughsoftware in an interactive set-up for better understanding of themovement.

According to one implementation, recording a three-dimensional map ofwhere the sample has traveled involves the use of a “coordinatedposition”. Typically, the stage has its own coordinate system on themicroscope. In some implementations, the Piezo may be in its owncoordinate system independent of the stage. The beam deflection isalmost always in its own coordinate system, often not represented in SIunits; for example, the beam deflection may be measured as a percentageor in DAC (digital to analog converter) units. Also, systems candigitally register the sample for the finest adjustments which needs tobe calculated into that coordinated position. However, there is nothingin the prior art that can link all the available positioners coordinatesystems into a “coordinated position” that combines the stage position,piezo position, beam position, and digital registration to give anabsolute position and vector for the sample of interest. Implementationsdisclosed herein overcome such limitations of the prior art.

The control system can capture the registered movement as a drift rateor a drift vector. The control system can subsequently generate a visualrepresentation of the drift rate or the drift vector to generate asingle coordinated position by combining a digital registration appliedto an image of the region of interest with at least one of an x-axis,y-axis, and z-axis coordinate planes. The visual representation of thedrift rate can be in the form a compass display, a bar display, anumerical value display, and/or a graph display. The control system canalso register the movement as a drift rate and further generate anormalization of the drift rate.

The control system can manipulate a template of an image of the regionof interest over a predetermined period of time to generate a currentmorphology or intensity profile. The control system can accordinglyutilize filtering techniques and frame averaging to morph the templatemore like the active region of interest to preserve history but react tomore dynamic samples. The control system is further configured toprovide a visual representation of a drift rate or vector associatedwith the registered movement. Typically, the stage coordinates areseparately tracked from piezo, separately tracked from beam position. Bycontrast, by combining all these coordinate planes with the digitalregistration applied to the image, the control system can allow for asingle “coordinated position” to be tracked in x, y and z coordinates oraxes. In at least one embodiment, the “coordinated position” may beseparated from the indicator noting the drift rate or drift vector. The“coordinated position” can be subsequently used by the control systemfor other purposes such as creating a particle tracking plot, creating a3d plot of where a feature went over time, and similar other plots.

Whereas during drift correction, it may be difficult to determine whenthe sample has stopped moving enough for a high-resolution acquisitionwith longer dwell time or exposure time, the control system as describedherein can conveniently overcome such shortcomings of the art. Toovercome such shortcomings, the control system can provide a visualrepresentation of drift rate; the control system can further normalizethis drift rate and display the same as an easy to read tool.Furthermore, the control system can provide for taking into a user'sselection of exposure time, magnification and other factors anddetermining a drift rate that is acceptable under such selections toachieve a high-resolution image. In one embodiment, the drift rate iscalculated from the vectors created from the “coordinated position”. Thecontrol system can further guide the user to either wait or adjust theimaging conditions required for the image quality desired.

The control system can be further configured to automatically choose oneor more of: a dwell rate and an exposure time to ensure a stable imageresulting from an in-situ stimulus being applied. For example, in caseswhere the user needs fast ramp rates and high resolution at a specificmagnification, the control system can provide for fast ramp rates anduse the slowest ramp rate that will enable successful tracking. Thecontrol system can further average frames on the digitally registeredsample to achieve the resolution. Regarding the coordinated positioncoordinates, typically, the stage coordinates are separately trackedfrom piezo, separately tracked from beam position. By combining allthese coordinate planes with the digital registration applied to theimage, a single “coordinated position” can be tracked in x, y, and zaxes.

The control system can provide for the capture of the performance of anin-situ holder and a MEMS sample support during the experiment. Thisperformance information can be obtained from both calibrated or“hard-coded” behavior, and further by constantly measuring actualperformance because MEMS sample supports differ from chip to chipslightly. This captured information can be used to further improvein-situ stimulus being applied to the region of interest, for example,in the form of drift vectors. The performance of each e-chip and holdercombination can be generally predicted by the control system asdescribed herein. It should be noted that the magnitude and exactdirection can vary quite a bit between e-chips and holders and may notbe completely captured in a single-time calibration. A certain amount ofon-the-fly learning of the performance of the experimental e-chip andholder could improve on the drift vectors, and the control system asdescribed herein can advantageously help improve the drift vectors.

In various embodiments, the control system disclosed herein isconfigured for sample tracking in an electron microscope. The controlsystem can comprise software instructions stored in a memory. Thesoftware can be stored in a non-transitory computer-readable mediumcapable of storing instructions. The instructions when executed by oneor more processors, can cause the one or more processors to perform oneor more of the tasks described herein. In one embodiment, the controlsystem can comprise a one or more instructions stored in anon-transitory computer-readable medium. The one or more instructionsthat, when executed by one or more processors, may cause the one or moreprocessors to register a movement associated with a region of interestlocated within an active area of a sample under observation with anelectron microscope, and direct an adjustment of the microscope controlcomponent to dynamically center and/or dynamically focus the viewthrough the electron microscope of the region of interest, wherein theadjustment comprises a magnitude element, and/or a direction element.

In one embodiment, the instructions can be accessed and executed by ageneral-purpose processor (GPU). In one embodiment, the softwareinstructions can be accessed and executed by a central processing unit(CPU) of a computing device. In one embodiment, the softwareinstructions associated with the control system can execute on a serverin communication with the internet. In one embodiment, a storagecomponent may store information and/or software related to the operationand use of control system. For example, the storage component mayinclude a hard disk (e.g., a magnetic disk, an optical disk, amagneto-optic disk, a solid state disk, etc.), a compact disc (CD), adigital versatile disc (DVD), a floppy disk, a cartridge, a magnetictape, and/or another type of computer-readable medium, along with acorresponding drive.

According to at least one embodiment, the control system includes aserver or a computing device that performs one or more processesdescribed herein. The server or the computing device may perform theseprocesses in response to a processor executing software instructionsstored by a non-transitory computer-readable medium, such as a memoryand/or storage component. A computer-readable medium is defined hereinas a non-transitory memory device. A memory device includes memory spacewithin a single physical storage device or memory space spread acrossmultiple physical storage devices. Software instructions may be readinto the memory and/or storage component from another computer-readablemedium or from another device via communication interface. Whenexecuted, software instructions stored in the memory and/or the storagecomponent may cause the processor to perform one or more processesdescribed herein. Additionally, or alternatively, hardwired circuitrymay be used in place of or in combination with software instructions toperform one or more processes described herein. Thus, implementationsdescribed herein are not limited to any specific combination of hardwarecircuitry and software.

According to at least one embodiment, the control system comprises amemory and a processor. The control system is configured to registermovement associated with a region of interest located within an activearea of a sample under observation, the region of interest positionedwithin a field of view of an electron microscope. The registeredmovement includes at least one of an x-axis, a y-axis, and a z-axiscomponent. The control system is further configured to adjust anelectron microscope control component to dynamically center and/ordynamically focus a view through the electron microscope of the regionof interest. The control system determines a magnitude of the adjustmentand/or a direction of the adjustment based on the registered movement.

Embodiments described herein can provide for keeping a region ofinterest stable and in the field of view regardless of stimulus to thesample. Additionally, embodiments of the presently disclosed subjectmatter can provide for a novel technique for quickly and easilyquantifying beam effects and other microscope parameters on a givensample under study to establish safe limits on such beam effects andother microscope parameters prior to further imaging of the sample understudy. Embodiments can advantageously provide for event triggering aswell for measuring, displaying, and limiting microscope parametersapplied to a sample. Embodiments disclosed herein can further provide anautomatic beam unwinding process. Embodiments disclosed herein can alsoprovide for a combination of measuring dose and beam blanking specificlocations when a threshold is reached. Embodiments disclosed herein canfurther provide for combining autofocus/auto centering with tomography.Embodiments can provide for automated feature tracking, event triggeringas well as measuring, displaying, and limiting microscope parameters ofa sample in an electron microscope undergoing in-situ environmentalchanges. Further, embodiments of the presently disclosed subject mattercan correct for thermal drift and other physical movements common toin-situ studies in an electron microscope through software. Embodimentsof the presently disclosed subject matter can use image analysis,in-situ measurements, or microscope behavior to trigger changes to themicroscope or in-situ environment through software. Embodiments of thepresently disclosed subject matter can track dose, dose rate, andin-situ stimulus applied to a feature and the use of a single ormultiple regions of interest to compare the relative impact of beamdamage or in-situ stimulus for a stable or moving system.

The control system can include software that combines analysis of userspecified regions of interest, background drift and predictive behaviorto track features in the electron microscope often at the atomic scale,then commands positioners in the electron microscope to center and focusthe region of interest. According to one or more embodiments, thecontrol system registers movement at a nanoscale or an atomic scale. Itcan also be at the micron scale at lower magnifications.

According to at least one embodiment, a control system configured forsample tracking in an electron microscope environment includes at leasta memory, a processor, and a microscope control component. The controlsystem is configured to register a movement associated with a region ofinterest located within an active area of a sample under observationwith an electron microscope. The registered movement includes at leastone or more directional constituents including an x-axis constituent, ay-axis constituent, and a z-axis constituent. The region of interest ispositioned within a field of view of the electron microscope. Inresponse to the registered movement, the control system is configured todirect an adjustment of the electron microscope control component todynamically center a view through the electron microscope of the regionof interest, and/or dynamically focus the view through the electronmicroscope of the region of interest. The adjustment can include amagnitude element and/or a direction element. In some embodiments, theadjustment of the microscope control component comprises one or more of:an electron beam deflection, and a focal plane adjustment.

In some embodiments, the registered movement includes at least one of analpha-tilt and a beta-tilt. The control system can counteract theregistered movement in the form of a alpha-tilt and/or a beta-tilt bydirecting an adjustment of an electron microscope control component todynamically center a view through the electron microscope of the regionof interest, and/or dynamically focus the view through the electronmicroscope of the region of interest. The adjustment comprises amagnitude element, and/or a direction element.

The control system is configured to adjust the electron microscopecontrol component to counteract the registered movement relating tophysical drift, thermal drift, and/or electrical drift imparted by theelectron microscope. The control system is also configured to adjust theelectron microscope control component to counteract the registeredmovement relating to an alpha tilt of a beam of the electron microscopeand a beta tilt of a beam of the electron microscope. The control systemis also configured to adjust one or more electron microscope controlcomponents to counteract the registered movement relating to a driftoccurring from a sample holder settling into a new location after astage movement. The control system can further adjust the electronmicroscope control component to counteract the registered movementrelating to a thermal settling not related to an in-situ stimulus. Thecontrol system is also configured to adjust the electron microscopecontrol component(s) to counteract the registered movement caused by oneor more of: mechanically deforming, altering an acceleration voltageapplied to, electrically probing, heating, cooling, and imaging of, thesample in a gas or fluidic environment. The control system can furtheradjust the electron microscope control component to counteract theregistered movement caused by in one or more of: pressure, flowrate, anda constituent, in an environment contiguous to the sample.

The control system is also configured to adjust the electron microscopecontrol component to counteract the registered movement caused by driftfrom the physical positioning systems of the microscope or samplesupport. The control system is also configured to adjust the electronmicroscope control component to counteract the registered movementcaused by the holder physically settling into a new position aftermoving the mechanical stage. The control system is also configured toadjust the electron microscope control component to counteract theregistered movement caused by the drift from thermal equalization of thesample support stemming from difference in temperature between theexternal room and the sample location inside the column. The controlsystem is also configured to adjust the electron microscope controlcomponent to counteract the registered movement caused by thermal orelectrical drift from optics adjustments. The control system is alsoconfigured to adjust the electron microscope control component tocounteract the registered movement caused by one or more of: a change inacceleration voltage of the gun, a power change in a corrector, a powerchange in another component of the optics. The control system is alsoconfigured to adjust the electron microscope control component tocounteract the registered movement caused by drift in the x-axis andy-axis created during small tilt or tomography sequences. The controlsystem is also configured to adjust the electron microscope controlcomponent to counteract the registered movement caused by a backgrounddrift within the active area.

The control system is accordingly configured to adjust the electronmicroscope control component to counteract the registered movementrelating to one or more of: in-situ stimulus applied to the sample,change in an environmental condition in an area contiguous to thesample, physical drift imparted by the microscope, physical driftimparted by a sample support positioning system of the microscope,thermal equalization occurring on the sample support, thermal drift ofan electron microscope optics, thermal drift of an electron microscopegun, electrical drift of the electron microscope optics, and electricaldrift of the electron microscope gun. The control system is furtherconfigured to apply an in-situ stimulus to the region of interest,wherein the adjustment comprises a drift correction along an x-axis anda y-axis.

In at least one embodiment, the control system is further configured toapply an in-situ correction (or in-situ stimulus) to the region ofinterest, wherein the adjustment/correction/stimulus comprises a driftcorrection along the x-axis, y-axis and/or z-axis. In at least oneembodiment, the microscope control component is in electroniccommunication with various components of an electron microscope such,for example, a mechanical stage, a goniometer, a piezo component of thestage, an illumination of an electron beam, a projection of the electronbeam, electromagnetic deflection of the electron beam, and a movement ofthe electron beam. In at least one embodiment, the control system isalso configured to register the movement at a micron scale, a nanometerscale, or an atomic scale. In at least one embodiment, the controlsystem is also configured to simultaneously register movement associatedwith a plurality of regions of interest located in the sample underobservation. In at least one embodiment, the control system is alsoconfigured to register the movement by referencing a template image ofthe region of interest against a remainder of the active area of thesample. In at least one embodiment, the control system is alsoconfigured to manipulate a template image of the region of interest overa predetermined period of time to generate a current morphology profileor a current intensity profile. It is to be noted that the template thatthe correction algorithm references for corrections is not a staticsnapshot of the sample from a while ago; instead, the template isconstantly morphed through image filters so that morphology andintensity profile is more similar to features of the sample that makesup the region of interest. In at least one embodiment, the controlsystem is also configured to capture the registered movement as a driftvector associated with one or more of: a structure of interest, a regionof interest, and a background region, of the sample under observation.

In at least one embodiment, the control system is also configured toalert a user when the registered movement is below a predetermined rate.Alerting the user when a registered movement is low can be beneficial tomake the user aware of when a high-resolution image is ready to becaptured.

In one embodiment, the control system is also configured to improveaccuracy of the drift vector by applying performance data related to asample holder and/or a MEMS sample support to the drift vector. Thecontrol system can also analyze the drift vector to predict or select afurther region of interest for observation. The control system canfurther apply an in-situ stimulus to the region of interest. The in-situstimulus can be in the form a drift vector generated by the controlsystem based on the movement registered at the region of interest. Thecontrol system applies the generated drift vector to a further area ofinterest within the sample. The control system can also compare thedrift vector with a reference template image of the region of interestto identify a change that has occurred to the sample under observation.

In one embodiment, the control system is further configured toautomatically identify a new region of interest in response to at leastone of the following: a field of view (FOV) change, a sample change, amicroscope status update, an un-blanking of an electron beam, an openingof a column valve, a screen raising, and an imaging condition change.The control system is further configured to digitally delineate theregion of interest from a live image stream of the field of viewdisplayed on a graphical user interface by one or more of: marking acontour on a live image stream of the field of view displayed on agraphical user interface; marking a shape on a live image stream of thefield of view displayed on a graphical user interface; superimposing apre-existing shape on a live image stream of the field of view displayedon a graphical user interface; capturing a double-click event performedon an area within a live image stream of the field of view of theelectron microscope displayed on a graphical user interface; andcapturing a click and drag event on an area within a live image streamof the field of view of the electron microscope displayed on a graphicaluser interface. In one implementation, the control system is furtherconfigured to apply a centering motion to the region of interest whenthe control system determines that the region of interest has moved awayfrom a center of the field of view or from a reference point within thefield of view. The control system can further determine an in-situstimulus to be applied in real time based on one or more of: a driftvelocity detected in the registered movement, and a detected imagingcondition of the region of interest, a performance parameter of a samplesupport; and a performance parameter of a sample holder. The controlsystem is further configured to determine an in-situ stimulus to beapplied in real time based on one or more of a drift velocity, a driftspeed, and a drift resolution detected in the registered movement. Thedetected imaging condition of the region of interest comprises one ormore of: a magnification level, and an image acquisition time. Thecontrol system is further configured to counteract the registeredmovement by one or more of: applying a physical adjustment, applying adigital adjustment, filtering an image displayed in a live image streamof the field of view displayed on a graphical user interface, andfiltering an image displayed in a drift corrected image sequence.

In various embodiments, the control system is further configured todirect generation of a seamless video of the region of interest. Thecontrol system can also digitally correct an image of the region ofinterest. In one implementation, while the image of the region ofinterest is corrected by the control system, an image of the remainingarea of field of view is not digitally corrected. In one embodiment, thecontrol system is further configured to enable a user to specify apredetermined quantity of digital correction to be applied to the atleast one image of the region of interest before application of aphysical correction to the at least one image of the region of interestis triggered. In one implementation, an image of a total area of thefield of view is not corrected. The digital correcting can include anyof the following techniques: digitally shifting the image, digitallycropping the image, digitally blurring the image, digitally sharpeningthe image, digitally adding to edges of the image, digitally addingbackground pixels to the image, and digitally adding foreground pixelsto the image. The control system can also save a digital corrected copyof the image, and a regular uncorrected copy of the image. In someembodiments, the control system further comprises a review utility,wherein the review utility is configured for reviewing a captured imageor a captured video indexed with one or more of: a microscope metadata,an in-situ metadata, and an imaging condition. This can advantageouslyprovide for the ability to scrub through images after an experiment. Thereview utility can be configured to generate a mathematical algorithmfor application to one or more of: the image, the microscope metadata,the in-situ metadata, and the imaging condition. The mathematicalalgorithm can be applied to a drift corrected sequence of images,wherein the control system is further configured to evaluate a change inthe adjustment applied over a predetermined time interval. Themathematical algorithm can comprise at least one of: a transformanalysis, an intensity plot, a pixel intensity statistic, acrystallinity score, a focal score, a variance score, a contrast score,a particle size analysis, and a distance between points analysis.Accordingly, a drift corrected sequence can allow a user to see how aparticle or sample changed over time; the user can quantify this bydragging math across frames of a drift corrected sequence. The controlsystem is further configured to export a predetermined sequence ofimages reviewed by the control system to a permanent disk space in apredetermined image format. The control system is further configured toapply the mathematical algorithm to an image or a metadata to isolate apredetermined sequence of images or to export a predetermined sequenceof images. For example, the control system may isolate only the imagesin good focus or isolate when the correlation against the templatechanged by a predetermined amount, or isolate only the images when thetemperature was changing between two predetermined outer limit values.

The control system can also generate a video based on one or more of:consecutive digitally corrected images, and consecutive digitallyuncorrected images. In at least embodiment, the video can comprise adigitally corrected ultra-stable movie of the region of interest. Invarious embodiments, the control system generates a video based onconsecutive images by applying various techniques such as, for example,a transform analysis such as FFT and CTF, an intensity plot, a pixelintensity statistic, a focal algorithm analysis, a brightnessadjustment, a contrast adjustment, a gamma adjustment, a metadataoverlay layer, and a shape overlay layer. In one embodiment, the videocurated by the control system comprises a digitally uncorrected movie ofthe region of interest. In one embodiment, the video curated by thecontrol system comprises a digitally corrected stable movie of theregion of interest.

In various embodiments, the control system is further configured todevelop a focus score of a focus level of the region of interest byanalyzing a Fast Fourier Transform (FFT) value associated with an imageof the region of interest. The control system can also develop a focusscore of a focus level of a further region of interest located withinthe active area by analyzing a variance of pixel intensities in an imageof the region of interest. The control system can also develop a focusscore that quantifies contrast, normalized variance, gradient andsimilar other parameters. The control system is further capture an outof focus image of the region of interest to calculate an optimal z-axisdistance of the sample from a lens of the electron microscope, whereinthe z-axis is perpendicular to a plane corresponding to the region ofinterest. The x-axis as mentioned herein can be parallel to a bottom orlower edge of the plane corresponding to the region of interest, whereasthe y-axis as mentioned herein can be parallel to a side edge of a planecorresponding to the region of interest. For example, assuming the planecorresponding to the region of interest to represent a rectangle shape,the x-axis may be parallel to the top and bottom edges of the rectanglewhile the y-axis may be parallel to the left side edge and right sideedge of the rectangle. The control system can further continuouslymonitor a focus level of the region of interest. The control system cangenerate a normalized focus score based on the focus level. The controlsystem can further generate a normalized focus score based on a focalquality analysis and physically aligned images. The control system canfurther generate a normalized focus score based on a focal qualityanalysis and digitally aligned images. The control system is configuredto change a focus level of the region of interest by applying a driftcorrection along a z-axis, wherein the z-axis is perpendicular to aplane corresponding to the region of interest. The control system candisplay a focus score on a graphical user display, wherein the focusscore is juxtaposed with a display of a predefined focus score. Thecontrol system can manipulate a focus level to an over-focus conditionor an under-focus condition. The control system can further use a focuscontrol algorithm to continuously adjust an objective lens of theelectron microscope to generate a normalized focus score.

The change to the sample under observation can represent any kind ofchange in the status quo include aspects such as a phase change, aprecipitate formation, a morphology change, a reaction with asurrounding environment, a reaction with a nearby element, and acoalescing occurring within the sample under observation. The controlsystem can register the movement as a registration algorithm and/or analignment algorithm. The control system is further configured tocalibrate the registration algorithm and/or the alignment algorithm.

In some embodiments, the control system is further configured toregister the movement as a pixel shift and translate the pixel shiftinto a correction distance for a positioner of the electron microscope.The control system can also operate to translate a plurality of thepixel shifts into a drift velocity vector and/or a drift accelerationvector. Accordingly, the control system is further configured to a applya correction distance to the positioner only when the resolution of thepositioner can support a magnitude of the correction distance. Thecontrol system is also configured to apply a correction distance to thepositioner such as to maximize a frame rate of a resulting driftcorrected sequence. A plurality of pixel shifts is preferred so thatphysical movements are scheduled only when the resolution of the desiredpositioner can support the magnitude of the required move. A pluralityof pixel shifts is also preferred so that physical movements areschedule only in opportune moments since the resulting positioner movecould temporarily blur the view when moved mid-capture. Further, aplurality of pixel shifts is preferred so that the frame rate of theresulting drift corrected sequence is as high as possible. Users oftendecide to skip frames during physical movements to remove the residualeffect of the move from calculations and the drift corrected sequence.Users generally do not need to skip frames when the drift correction isonly a pixel shift. In response to a movement registered by the controlsystem, the control system can trigger various actions such as, forexample, pausing an in-situ stimulus, holding constant the in-situstimulus, and changing a ramp rate of the in-situ stimulus, amongothers.

The control system can include algorithms to perform tasks such asreducing a size of a move as the normalized focus score approachescloser to a best registered focus score. The control system can furtherinclude algorithms to perform tasks such as increasing the size of themove as the normalized focus score deviates away from the bestregistered focus score. The algorithms of the control system are alsoable to or configured to tune re-focus points of the lens of theelectron microscope, wherein the re-focus points define a focus envelopeby manipulating an indicator handle. The control system also includes az-axis focus control that can include aspects such as a beam control, apiezo control, and a stage control. The control system is furtherconfigured to perform a calibration of a camera parameter, a detector.Calibrations operate to improve performance of the drift correction andto insure accurate moves regardless of the application. For example, thecontrol system can be configured to perform a calibration of one or moreof: a camera parameter, a detector parameter, a positioner parameter,and an in-situ control parameter. The calibration can comprise arotational offset, and a magnification focus envelope, among others. Itis to be noted that a microscope profile is mostly rotational offset,focus step sizes, positioner capabilities and network setup. The controlsystem can store a calibration value associated with the calibration ina calibration database, and compare a measured value against thecalibrated value on a periodic basis; the control system can alsomonitor performance of the control system against one or morecalibration values. The control system can also run the calibrationduring each movement registering session.

In at least one embodiment, the calibration value corresponds to apositioner. The calibration value is generated for at least one of: abacklash, a movement limit, a movement timing, a resolution, a totalrange, a preferred range, a hysteresis, a minimum move time period, aunit conversion, a neutral position, and a minimum move time periodassociated with the positioner. In one embodiment, the calibration valuecorresponds to a holder, wherein the calibration value is associatedwith one or more of: an imaging origin adjustment, a x-axis adjustment,a y-axis adjustment, and a z-axis adjustment, wherein the z-axis isperpendicular to a plane corresponding to the region of interest. In oneembodiment, the calibration value is associated with a change in one ormore of: a pressure, a flowrate, and a mechanical deformation,associated with the sample. In one embodiment, the calibration value isassociated with an expected movement model corresponding to a heatingholder or cooling holder. In one embodiment, the calibration value isassociated with an expected movement model corresponding to one or moreof: a drift velocity relating to a change in unit temperature, a coolingramp-rate, and a heating ramp-rate.

In some embodiments, the control system is configured to apply thecalibration value to an in-situ control input that comprises one or moreof: a current value, a temperature set point, and a fluid flow rate. Insome embodiments, the control system is also configured to calculate amaximum thermal ramp-rate achievable during a concurrent application ofan in-situ stimulus and a drift correction adjustment. The adjustmentcan also be in the form of a drift correction applied along a z-axis tocompensate for an anticipated movement of a membrane associated with thesample under observation, wherein the z-axis is perpendicular to a planecorresponding to the region of interest, wherein a x-axis and a y-axisare parallel to the plane of the region of interest. The adjustment caninclude a drift correction, wherein the control system is furtherconfigured to pause applying the drift correction when at least one ofan x-axis parameter and a y-axis parameter of a positioner falls outsideof a predetermined range. The adjustment can comprise a drift correctionapplied along a z-axis to compensate for an anticipated movement of amembrane associated with the sample under observation, wherein thez-axis is perpendicular to a plane corresponding to the region ofinterest, wherein a x-axis and a y-axis are parallel to the plane of theregion of interest.

In various embodiments, the control system can calculate the maximumthermal ramp-rate achievable using one or more of: a ratio of an area ofa field of view relative to an area of the region of interest, apositioner timing, an image update rate, and an expected drift rate. Thecontrol system can also alter a thermal ramp-rate affecting the regionof interest in response to a change in a refresh rate of an image of theregion of interest. The control system can further decrease or pause athermal ramp-rate affecting the region of interest in response to a userattempting to manually bring a second region of interest into focus.

The control system is further configured to display, on a graphical userdisplay device, an electron microscope control and a drift correctionparameter applied to the region of interest in a same single userinterface. The control system is also configured to display, on agraphical user display device, an impact of one or more of: amagnification value, an active detector size, a pixel resolution, abinning, a dwell rate, and an exposure time, for evaluating aneffectiveness of an in-situ stimulus applied to the region of interest.The control system is additionally configured to assist a user inprioritizing one or more of: a camera option, a detector option, anelectron microscope set-up feature, and an in-situ stimulus, forgenerating a stable image resulting from an in-situ stimulus applied tothe region of interest. The control system can automatically choose adwell rate and an exposure time to ensure a stable image resulting froman in-situ stimulus applied to the region of interest. The controlsystem can further automatically adjust an in-situ stimulus applied tothe region of interest in response to a user adjusting one or more of: apixel resolution, a magnification value, and a thermal ramp-rateassociated with the electron microscope. The control system can alsopredict a movement associated with a further region of interest based onthe movement registered at the region of interest.

In at least one embodiment, the control system is configured to set atrigger function to an in-situ stimulus applied to the region ofinterest, wherein the trigger function is activated when a change isobserved to at least one of: a sample feature, an electron microscopecondition, an in-situ stimulus source, and an in-situ stimulus reading.In one embodiment, the adjustment of the microscope control componentcomprises a trigger function that is activated when a change is observedto a sample feature, an electron microscope condition, an in-situstimulus source, or an in-situ stimulus reading. In at least onembodiment, the trigger function adjusts a parameter affecting at leastone of: the electron microscope, a camera associated with the electronmicroscope, and a detector associated with the electron microscope. Insome embodiments, the control system can turn a detector associated withthe electron microscope on or off when a sample temperature fallsoutside of a predetermined range.

In some embodiments, the control system further comprises a userinterface configured for developing the trigger function. In someembodiments, the control system is further configured to allow a user toset an electron dose rate limit for the sample under observation. Insome embodiments, the control system is also configured to calculate anelectron dose rate for the electron microscope as a function of aposition of an electron microscope lens and time. In some embodiments,the control system also monitors to ensure that the electron dose ratedoes not exceed a predetermined electron dose rate limit. The controlsystem can further set limits on a cumulative electron dose, in additionto limits on an electron dose rate.

In at least one embodiment, the control system is configured to display,on a graphical user display device, an image of an electron dose rate ina heatmap form; the control system is further configured to display, ona graphical user display device, an image of a cumulative electron dosein a heatmap form; the control system is configured to automaticallyadjust the displayed image to counteract a change in one or more of asample position and a magnification level. The control system can alsogenerate an automated report based on the registered movement and theapplied in-situ stimulus. The control system can allow a user to set asafety limit to prevent irreversible damage to the sample. The controlsystem can further measure an impact of an electron beam on one or moreof: a sample shape, a sample composition, a sample density, and anelectrical characteristic of the sample. The control system canadditionally record the registered movement over a period time togenerate a three-dimensional map of a history of movements occurring inthe region of interest. The control system can also provide a visualdisplay of the history of movements in a three-dimensional path on agraphical user display device. In some embodiments, the visual displayof the history of movements is rotatable in an interactive manner inresponse to a user prompt. In some embodiments, the control system cancalculate a maximum permissible movement based on one or more of anacquisition rate (e.g., exposure time in TEM mode and dwell time in STEMmode), and a magnification level, as selected by a user. The controlsystem can further guide the user to adjust an imaging condition toprevent reaching the maximum permissible movement. The control system isalso configured to set a trigger function associated with auxiliarydevices such as a mass spectrometry device coupled to the electronmicroscope, a gas chromatography device coupled to the electronmicroscope, and a liquid chromatography device coupled to the electronmicroscope.

In at least one embodiment, the control system can adjust anenvironmental condition associated with the sample in response to thetrigger function being activated by the control system. The controlsystem can further adjust an environmental condition associated with thesample when a measured concentration of a substance contained incirculating water exiting an in-situ holder coupled to the electronmicroscope falls outside of a predetermined range. The control systemcan further display, on a graphical user display device, a listing ofimages of portions of the sample previously observed by a user alongwith a dose or a dose rate associated with each listed image. Thecontrol system is further configured to display, on a graphical userdisplay device, a listing of images of portions of the sample exposed toa predefined level of electron radiation from an electron beam of theelectron microscope.

In various embodiments, the control system is further configured tocontinuously monitor aspects such as a field of view of the electronmicroscope; x-axis, y-axis or z-axis parameters of at least onepositioner associated with the electron microscope; a z-axis parameterof at least one positioner associated with the electron microscope; analpha tilt of a holder; a beta tilt of the holder; an image refreshrate; a beam blanker state; a column valves state; a screen angle; amicroscope metadata; and, an imaging system metadata.

In some embodiments, the applied in-situ stimulus comprises moving apositioner, wherein the control system is further configured to choosethe positioner from one or more of: a stage positioner, a piezopositioner, and a beam positioner. The control system is configured tocalculate a time required to move the positioner to minimize impact of amovement of the positioner on a saved image sequence. The control systemcan further select the positioner based on the magnitude of the appliedin-situ stimulus. The control system can additionally select thepositioner based on an amount of the applied in-situ stimulus remainingto reach a predetermined maximum magnitude of the applied in-situstimulus. The control system can zero out a further in-situ stimulusthat was previously applied to the positioner. The control system canalso assign one or more automatic limits to an electron beam position ofthe electron microscope to prevent or reduce stigmation. The controlsystem can further permit a user to toggle between the region ofinterest and the further region of interest. The control system caninitiate acquisition of high-resolution images of the region of interestwhen the registered movement is below a predetermined value orpredetermined rate.

In at least one embodiment, the control system is further configured toidentify a user-initiated action when it detects a movement associatedwith at least one of: a x-axis position of a mechanical stage, a y-axisposition of the mechanical stage, a z-axis position of the mechanicalstage, a piezo stage deflection, a beam deflection, a piezo stage, afocal plane, an alpha tilt, a beta tilt, an image refresh rate, and animaging condition. The control system can also calibrate or trigger anin-situ stimulus based on the user-initiated action. The control systemcan further pause or halt an in-situ stimulus that conflicts with theuser-initiated action.

According to various embodiments, registering sample movement can beaccomplished by the control system by template matching a subset of theimage, usually the primary region of interest, against the rest of thefield of view. Techniques to reduce the large amount of“salt-and-pepper” or background noise common in TEM (transmissionelectron microscopy) and STEM (scanning transmission electronmicroscopy) image sequences, such as a median blur filtering improvesregistration and alignment algorithms. It can further include filteringtechniques. Registering a pixel shift can then be translated into acorrection distance for positioners associated with the electronmicroscope. A combination of these pixel shifts can be translated into adrift velocity vector and a drift acceleration vector.

The control system can permit a user to select one or more primaryregions of interest by selecting them from the live image stream in thesoftware, for example, by making the selection of an interactivegraphical user display coupled to the control system. The selection ofregions of interest could be done by drawing a contour/border on theimage, drawing a shape on the image, or by picking from one of thepredetermined shapes. The control system can further provide for easyresizing. There could be multiple regions of interest including, forexample, one for x, y drift correction, and one for z auto-focus. Thecontrol system as described herein can provide for the x, y centeringregion of interest to be in the center of the field of view, thusenabling users to easily move key features to the center beforeinitiating drift correction will help. The control system as describedherein can provide for accomplishing this by double clicking on theimage. Alternatively, the control system as described herein can providefor accomplishing this by applying a centering motion to a position thatis not at the center of the field of view. Once drift correction isinitiated, new regions of interest could be set through the software,which would update any reference templates. This could be accomplishedby double clicking on a new region or drawing a new region of interest.

In some embodiments, the control system is configured to reduce oreliminate the movement to facilitate generation of a seamless video ofthe region of interest by applying a physical adjustment, applying adigital adjustment, filtering an image displayed in a live view, and/orfiltering an image displayed in a drift corrected image sequence. Thesystem can reduce or eliminate movement for the seamless video live byphysically correcting, digitally correcting, but also automaticallyfiltering the images displayed in the live view and drift correctedimage sequences. For example, the system can allow for skipping ofimages in the live view where the system is physically moving one of thepositioners eliminating these blurred images from the sequences. Thesystem can further send commanded movements to the positioners so thatthe blurred frames created by the positioners do not show up in thedrift corrected image sequence or live view. Knowing how long it takesto make a positioner move can provide the user with a seamlessexperience with only a few frames dropped or acquisition temporarilydelayed during the move. Accordingly, in various embodiments, thecontrol system is further configured to automatically skip one or moreblurred images to generate a drift corrected image sequence devoid ofthe one or more blurred images. The control system can furthercoordinate a timing of application of adjustment to synchronize with atime of acquisition of the one or more blurred images.

According to various embodiments, a region of interest's focus is scoredby the control system by analyzing the variance of pixel intensities inthe image. The control system can determine this through FFT (FastFourier Transform) calculation analysis, contrast transfer functionanalysis, and beam tilt analysis; the control system can alternatelydetermine this through deflections of the beam and by any other focalalgorithm. The control system can further operate to purposefully takethe image out of focus, both under and over, to help determine anoptimal Z height for the region of interest. However, this is notlimited to just lens and beam adjustments to bring the sample in and outof focus. The action taken by the control system is hierarchal in atleast one embodiment in that the control system will adjust the stage,beam and/or piezo depending on the scale of movement needed.

One procedure for changing samples (changing samples is very common inin-situ studies) involves the use of tunable filters to morph theoriginal registration template into the current live view. Additionally,this template can be completely reset in a strategical manner when userschange FOV, imaging conditions, or key items on the microscope. In atleast one embodiment, the control system is configured to manipulate atemplate of an image of the region of interest over a predeterminedperiod of time to generate a current morphology profile or a currentintensity profile. The control system can utilize filtering techniquesand frame averaging to morph the template more like the active region ofinterest; the control system can accordingly preserve history whilereacting to more dynamic samples. Accordingly, the control system canuse a template image for registering the movement. In some embodiments,the registered movement comprises a drift vector.

The control system can identify the time at which the sample ischanging, and based on the identification, the control system canadvantageously flag important events over long experiments with highframe rates; this can advantageously help in sorting key data from verylarge data sets and in saving images to file. This can furtheradvantageously help in pausing or holding and in-situ stimulus; this canadvantageously help in slowing ramp rates or in automatically updatingthe indicated region of interest.

According to at least one embodiment, changes to sample that controlsoftware could actively detect include the following:

1. Morphology related changes:

-   -   a. Surface faceting    -   b. Particle agglomeration/coalescence/sintering    -   c. Particle dissolving/etching/sublimation    -   d. Bulk—Inclusion dissolving/formation    -   e. Particle nucleation    -   f. Nucleation leading to sample growth

2. Phase related changes:

-   -   g. Kirkendall effect—void formation and outer shell formation    -   h. Crystallization/amorphization    -   i. Phase segregation    -   j. Grain boundary migration    -   k. Oxidation/reduction    -   l. Densification

3. Atomic changes:

-   -   m. Void/defect changes/dissipation/movement    -   n. Single atom dynamics    -   o. Zone axis determination    -   p. Graphene excitons

4. Automated features:

-   -   q. Detection of phase transformation    -   r. Detection of carbon contamination    -   s. Detection of liquid cell dewetting

In various embodiments, the control of positioners associated with theelectron microscope can be accomplished by one or more softwarealgorithms that form part of the control system. In some embodiments,the control of positioners can be hierarchal in that the control systemcan intelligently select the most appropriate correction option amongthe available correction options associated with the availablepositioners. The selection can be based on a combination of a driftvelocity and one or more imaging conditions such as a magnificationlevel and an image acquisition time. Common available positioners in theelectron microscope include mechanical stage control which is capable ofcoarsely moving the holder; in some examples, a piezo stage control isprovided for finely moving the holder; also controls may be provided forcontrolling the electron beam position through electromagneticdeflection of the electron beam of the electron microscope. Control ofthese positioners is often run through software; however, unlike thecontrol system as described herein, existing solutions do not tie suchcontrols to feature movement; also, unlike the control system asdescribed herein, existing solutions do not provide automated systemsfor continuous moves spanning all 3 positioners.

The control system can further reduce sample movement for seamlessvideo. The resulting image can then be digitally corrected by thecontrol system from the total field of view. The video could be of theFOV with the ROI centered showing how the ROI interacts with the rest ofthe sample. The control system can further provide for cropping orblurring of the perimeter pixels while keeping the region of interestcentered. The control system can further provide for saving both imagesets to file—the digitally corrected version and the uncorrectedversion. The control system can additionally provide for generatingvideos from consecutive images, digitally corrected for an ultra-stablemovie of the region of interest or uncorrected for the unaltered videofeed. Accordingly, embodiments of the presently disclosed subject mattercan perform these functions while simultaneously applying a physicalcorrection. The combination of these two functions can be beneficial.

The control system can further include capabilities for post-processinga perfect set of consecutive corrected images. For example, math oranalysis applied to an image can easily be applied to multiple imagessince they are physically and digitally aligned. Math and analysis caninclude transform analysis such as FFT and CTF, intensity plots, pixelintensity statistics, focal algorithm analysis, particle size analysis,particle distribution analysis, distance between two points,crystallinity analysis, resolution analysis, summing frames, averagingframes, image filters, brightness adjustments, contrast adjustments,gamma adjustments, metadata and shape overlay layers. By applyingmathematical functions or mathematical algorithms on a physically anddigitally aligned sequence of images, the control software can presenthow the sample changed over time quantifying the effects of theexperiment or imaging exposure. Additionally, mathematical functions ormathematical algorithms applied to the image can be used to sort andfilter images. Metadata can also be used to sort and filter images.Metadata can stem from imaging conditions, microscope conditions,in-situ data or calculations made on the image. For example, thesoftware can help identify only the images on a temperature ramp byanalyzing the sample temperature and then further limit the sequence toonly “in focus” images by filtering the focus quality score ornormalized focus quality score. Mathematical functions or mathematicalalgorithms can be applied to an image sequence after capture orprocessed live during image capture.

The control system is further configured to generate a video based onconsecutive uncorrected images. The control system includes capabilitiesfor post-processing a perfect set of consecutive corrected images. Forexample, math or analysis applied to one image can easily be applied tomultiple images since they are physically and digitally aligned. Mathand analysis can include transform analysis such as FFT and CTF,intensity plots, pixel intensity statistics, focal algorithm analysis,particle size analysis, particle distribution analysis, distance betweentwo points, crystallinity analysis, resolution analysis, summing frames,averaging frames, image filters, brightness adjustments, contrastadjustments, gamma adjustments, metadata and shape overlay layers.

In one embodiment, the control system as described herein can include(or be in the form of) a software suite provided by tradename AXONand/or by tradename Synchronicity. FIGS. 92 through 114 illustratevarious aspects of the AXON software suite (hereinafter referred to as“AXON system”, “AXON” or as the “control system” or simply “system”).The display of AXON on a digital display device such as a computermonitor can include three headings: “AXON Commands”, “MicroscopeCommands” and “Microscope Profile”. The “AXON Commands” and “MicroscopeCommands” section are used to feed the information in the “MicroscopeProfile” section that that characterizes a TEM column that the AXONsoftware suite is installed on or is otherwise electronically coupledthereto. “AXON Commands” include functions specific to the AXONapplication such as: “Reset Beam X/Y” that re-centers the beam to 0,0;“Reset Beam Z” that sets the defocus to 0; “Start Unwind Beam X/Y” thattriggers the X/Y unwind process (same process as lower indicator butwithout the restrictions); “Start Unwind Beam Z” that triggers the Zunwind process (same process as the lower indicator but without therestrictions); “Save Trace” that saves software diagnostic and traceinformation into a file; and, additional AXON specific commands toassist in service installation or diagnostics will be available in thissection as they are developed.

“Microscope Commands” include functions specific to the TEM such as:“Read Imaging Mode” that reads whether the system is operating in TEM orSTEM mode; “Read Magnification” that reads the magnification; “ReadPosition” which reads the current stage position for X, Y, Z, A and B(X, Y and Z corresponding to x, y and z axes; A representing alpha tiltand B representing beta tilt); “Set Position” that sets the stage to anabsolute coordinate for X, Y, Z, A and B; “Sync Position” that sets the“Set” positions to the current read position to assist in making smallstage increments; “Read Shift” that reads the current X, Y beampositions, which is TEM/STEM specific (TEM Shifts are often called“Image Shifts” whereas STEM Shifts are often called “Beam Shifts”;deflectors can be used for both types of movements); “Set Shift” thatsets the beam to an absolute coordinate in X, Y, which is TEM/STEMspecific; “Sync Shift” that sets the “Set” shifts to the current readposition to assist in making small beam shift increments; “Read Defocus”that reads the current Z beam position, often called the “defocus”value; “Set Defocus” that sets the Z beam position to an absolute value;and, “Sync Defocus” that sets the “Set” defocus to the current readposition to assist in making small defocus increments.

AXON can manage multiple microscope calibrations. Each TEM column canhave its profile automatically created by AXON when connected to theassociated microscope service. That connection can be first made throughthe service portal by clicking the “Test Connection” button against theavailable network microscope services. Upon successful connection, AXONcan create a microscope profile for that TEM populated with all defaultcapabilities. Performance can be enhanced by an accurate knowledge ofthe positioner and imager capabilities and the relationship between thetwo. While some fields can be manually entered after installation tests,several other field entries are based on automated procedures populatedat the end of the process.

“Microscope Profile” includes the microscope and all connected camerasand detectors are characterized on the system installation. The“Microscope Profile” can be a combination of automated and manualparameters calibrating the capabilities of each part of the column withrespect to the cameras/detectors. The microscope profile can be composedof data manually entered or automatically pulled from the connectedmicroscope, cameras, detectors, or in-situ systems. For example, the“Microscope Name” can be populated by the computer name of the TEMcolumn, and it can also be an editable field. The “Microscope Profilecan save networking and communication information such as the“Microscope Service Uri” which can be the uniform resource indicator tothe microscope service communication link and can include the “LastConnection Time” detailing the date/time of the last connection withthat microscope profile; “Positioner Capabilities” can be a header forall settings associated with the microscope's ability to move thesample; “Coordinate Transforms” can be a header for all X/Y rotationalalignment calibrations linking the positioners to the camera or detector(saved per detector, per positioner, per magnification); and, “FocusAssist Step Sizes” can be a header for all Z calibrations dictating thedistance it takes to bring a sample over, under and in focus dependingfor the imaging conditions and magnification (saved per detector, perpositioner, per convergence angle, per magnification).

As used herein, the following terms have the corresponding definitions.“Image Complete Threshold” is the percentage of unique pixels requiredto determine a new image during a continuous imaging stream. “ScanBoundary Acceptance Threshold” is the percentage of pixel rows from thebottom that the system attempts to target STEM scan boundaries beforedeclaring a unique image in a continuous imaging stream. “Range” is thephysical min and max limitations of the positioner as read by the columnAXON software in microns or degrees. Each positioner will have differentrange limits, and these can be different in the X, Y and Z plane as wellas alpha and beta tilt. “Preferred Range” is the preferred minimum andmaximum limitations of the positioner as read by the AXON software inmicrons or degrees. These can be the same as the range or could be asubset of the range. The preferred range can be used as a safety bufferor to prevent image degradation of the optics for the cases of beammovement. Each positioner may have a different preferred range, andthese can be different in the X, Y and Z plane as well as alpha and betatilt. The preferred ranged can be microscope dependent and/or OEM(original equipment manufacturer) dependent. “Resolution” is the minimummovement distance in microns that a positioner can be commanded throughthe AXON software after backlash has been accounted for. Each positionerwill have different resolutions, and these can be different in the X, Yand Z plane as well as alpha and beta tilt. “Hysteresis” is the distancein microns or degrees lost when changing direction on a givenpositioner. The hysteresis makes up the needed additional travel untilchanges in the resolution are discernable in the actual perceivedposition of the sample. Each positioner may have different hysteresisand can be different in the X, Y and Z plane as well as alpha and betatilt. These parameters may be used for making decisions on whether apositioner is the correct positioner for the magnitude of move requiredby the control software. “Min Move Time” is the time required for themove to complete and the image to settle for the smallest movedetermined by the resolution of that positioner. Each positioner willhave a different Min Move Time, and these can be different in the X, Yand Z plane as well as alpha tilt and beta tilt. “Move Pace” can be usedto quantify the additional scaling factor required for larger moves tocomplete and the image to settle, scaling linearly with the magnitude ofthe move. It is not required to break the movement time of a positionerinto both a minimum move time and a move pace, and these two parameterscan be summarized in a single movement time if preferred. “CoordinateTransforms” can be used to characterize the rational alignmentcalibrations linking the positioners to the camera or detector (savedper detector, per positioner, per magnification). The coordinatetransform process can be saved automatically after an automated processis triggered. An example of this process could be to move in 6 discretesteps for all relevant positioners accounting for hysteresis and savethe rotational alignment between the positioner and the active camera ordetector.

When a microscope calibration process is triggered, the system mayautomatically try to calibrate both the beam and stage for the camera ordetector with some exceptions. The system may only calibrate the STEMbeam when in STEM mode and the TEM beam when in TEM mode. Additionally,the process may only calibrate the beam when a certain subsection of thefield of view does not exceed the preferred range or physical range ofthe beam which can be dictated by the microscope profile. Likewise, thesystem may only calibrate the stage when the magnification is low enoughso that a certain subsection of the field of view does not exceed theresolution or hysteresis of the positioner.

When a positioner successfully finishes the calibration process, it maypopulate an entry under the “Coordinate Transforms” header detailing thecamera/detector, positioner, and magnification. The system may referencecalibrations in that order. On each move, the control system may lookfor a calibration for the correct camera or detector. If there is notone, it may alert the user that a calibration is needed. If there is, itmay reference the positioner capabilities to determine the correctpositioner based on the resolution and magnitude of required move. Ifthere is not a calibration for that positioner, it may alert the userthat a calibration is needed. If there is a calibration for thatpositioner, it may select the calibration associated with themagnification that the user is operating in or the closestmagnification.

In STEM mode, it may only be necessary to get a few calibrations, one atvery low magnifications for the stage, one at mid magnifications for thestage's smallest moves and the beam's largest moves, and then one athigh magnifications for the beam's smallest moves. In TEM mode, it maybe necessary to get more calibrations at multiple magnifications. It isnot uncommon for TEM cameras to rotate the image as new lenses areenabled.

“Focus Assist Step Size” is a header for all Z calibrations thatdictates the distance it takes to bring a sample over, under and infocus depending for the imaging conditions and magnification. Much likethe “Coordinate Transforms”, “Focus Assist Step Sizes” can be saved percamera/detector, per convergence angle, per magnification. Thesecalibrations can also be an automated process which steps the defocus inboth directions outward from the starting position in increasingmagnitudes until it reaches a prescribed limit. The prescribed limit canbe a fixed value or settings such as the “Calibration Maximum Step Size(um)” setting or the “Calibration Minimum Focus Quotient” setting. Toimprove the calibrations, if at any time, the control system gets abetter focus score (alternately referred to as a score of a focal level)while stepping outward, it may restart the process from the newposition. At the end of the process, it may bring the defocus back tothe best focus position and populate an entry into the “Focus AssistStep Sizes”. These entries apply a function to the points to help thecontrol system determine the size of step needed as a sample goes in orout of focus.

The control system is further configured to continuously monitor a focuslevel of the region of interest, and to use physically and digitallyaligned images along with focal quality analysis to enable a normalizedfocus score. Whereas focus scoring on a single image is important, butsince they are all physically and digitally aligned, a focus history canbe built by the control system based on the same set of features.Comparing the focus quality scores applied to a single frame againstwhat is possible can advantageously normalize the focus score. Anormalized focus score, in turn, can enable live analysis of focus toimprove or depict focus quality. The focus control algorithm of thecontrol system can constantly adjust the objective lens (defocus). Asthe normalized focus score approaches closer to the best registeredfocus score, the size of moves gets smaller (close to 0 nm). As thenormalized focus score gets worse, the adjustment size increases. Thedirection of move is tied to analysis of the normalized score history.Movements that result in a lower normalized score get factored into acontroller directed by the control system, with the controllerconfigured to eventually reverse the direction of move. The normalizedfocus score references a best possible focus. That The normalized focusscore and be updated on any new template (any time the imagingconditions change, FOV changes, etc.) and the template is morphed overtime through filters (such as bump filter) to account for morphologychanges or intensity profiles that may make a best possible focus nolonger attainable. The normalized focus score filtered for noise tocurtail the reaction of the controller to the noise inherent to EMimages. Since there may not be adequate history available on how well aprofile applies to different types of samples or other imagingconditions, this process can be triggered by users anytime driftcorrection is running. It can serve as an “auto focus” function to bringan out of sample back into focus faster and a calibration function tocalibrate the control system for that type of sample. All calibrationsare saved so this is not a necessary step on each experiment—onlyreserved in case the default behavior is not preferred. Drift correctiondoes need to be running for the focus assist calibration to guaranteethe control system is looking at the same region of interest through thecalibration.

A key step in AXON is to start a session. This sets the defaultoverlays, workflow and prioritizes connection type. Users can change thesession name to help organize data.

On installation, AXON can create a directory of support files organizedinto a predetermined folder directory present on a server. In thisdirectory, users can manually access files used by the application. AXONcan automatically create a log on each microscope connection orconnection with Clarity products. In one embodiment, the control systemas described herein can include a software suite provided by tradenameClarity (hereinafter referred to as “Clarity” or “control system” orsimply “system”). Accessing these logs can help determine how often andwhy users are using the AXON application.

The control system may create a folder for each session, separating the“Drift Corrected”, “Raw”, “Templates” and “Single Acquires” per session.This directory can be setup for first in, first out as the buffer sizeapproaches its maximum limit. The session folders may persist for aslong as there are images of that session still in the buffer. The imagescan be manually moved from this folder or exported using the AXONNotebook or any session or image review tool. As mentioned herein, AXONNotebook may refer to a tradename given to an image review tool formingpart of the control system according to one or more embodiments of thepresently disclosed subject matter. Each image can be saved with allrelevant metadata, however accessing this metadata may only be possiblethrough the AXON Notebook or supported review tools. These tools couldexport the images and export the metadata into a database or a CSV file.

AXON can rely on a microscope service and possibly additional cameraservices to interact with the TEM and cameras. These services areinstalled and run on the column and camera computers and communicatewith the AXON application. These services can be Microsoft windowsseries, formerly known as NT series, and enable long-running executableapplications that run in their own Windows session, but they can also bestandalone applications. These microscope services work well as along-running application that does not interfere with other usersworking on the same computer. On installation, a background service isstarted, and an icon can be created. That icon can indicate connectionstatus with AXON. It can be in a standby state until triggered by AXONthrough a “Connect” function; it then attempts to reach the TEM OS andimaging OS. On clicking this icon, a small lightweight UI for themicroscope service can be viewed. This application can have multiplepanes, opening up to panes such as “Status”, but easily toggleable to“Diagnostics” and “About”. Once connected to AXON, the Connect statusunder AXON may change state from “Not Connected” to “Connected”. Onceconnected to the microscope, the connection status under “Microscope”may change state from “Not Connected” to “Connected”.

In terms of image monitoring, AXON does not need to create the imagingsession or update conditions. The user can continue to setup the imagingconditions within their native imaging environment and AXON identifiesunique images through the image monitoring process managed within themicroscope or camera services. AXON polls the images as fast as it canscript the imaging service. Once the control system determines that theimage is unique, the process compiles the intensities of each pixel intoa bitmap with all associated metadata. The control system then sendsthat package from the microscope service to the AXON main application.Once the package is sent, the process commands any change to the TEMcolumn if needed like positioner updates. However the functions andfeatures of AXON is not limited to only setting up the imaging sessionin the native imaging environment, an embodiment could include asoftware that enables control of the imaging setup.

AXON receives this bitmap package and applies the image monitoringprocess settings to scale the raw bitmap pixels to the user'spreferences. The unscaled bitmap is typically very flat and verydark—not very visible. AXON has a few image normalizations optionsavailable in the settings, where the user can choose between“Histogram”, “Min-Max” and “None”. “Histogram” is the default setting.The user can set the histogram lower fraction and the lower pixelintensity and the upper fraction and upper pixel value. Once normalized,the process runs the bitmap through any image processing needed. Inparallel with analysis, the process converts the bitmap into a losslessPNG or any other file type for storage in the image buffer. Only thescaled image is converted, and the original bitmap is lost.

AXON can work with full resolution images but may bin the images downfor computation. This architecture can allow for performing imageprocessing in a local environment where one can leverage third partylibraries like OpenCV. This process works for single acquisitions,continuous acquisitions, STEM and TEM mode. It does require that theuser setup the imaging session in their native imaging environmentthrough either a “Search”, “View”, “Preview” or “Acquire”. There arecases where a connection is made, but images are not displayed in theAXON software. In these cases, AXON alerts the user with a dialoguestating why images are not being displayed. This is handled under thefollowing cases: column valves closed; beam blanked; and, screen down.Drift control may, in some instances, include corrections for movementin the X/Y plane, but not changes in height or focus.

In terms of hierarchy of positioners, the AXON system is built on ahierarchy of positioners. Ultra-fine movements can be handled though adigital registration until they hit a threshold where a beam movement istriggered to unwind the digital registration. Eventually the beammovements are also unwound by triggering a movement of the stage. Thepiezo could be utilized on compatible TEM columns. An example of digitalregistration is shifting the pixels and cropping, blurring, or filteringthe edges of the field of view. By allowing a small percentage ofdigital registration, it enables the AXON software to provide a seamlesslive view of the sample without constantly triggering movements of theTEM beam or stage, keeping the regions of interest consistent andprevents image tearing and shadowing. Beam movements are differentbetween TEM and STEM mode and are the finest physical movement availablewithin the AXON software. Any physical move is made to center the samplewhich may reduce the amount of digital registration applied to theimage. As the beam moves further from the aligned position the imagequality suffers, overall contrast reduces, and edges have less gradientBeam shifts in TEM and STEM mode, if moved too far, may result in adegrading image. AXON can operate to unwind the beam through stage moveswhen the resolution of the stage and magnification allows. Unwinding thebeam can be triggered manually and automatically through the AXONsoftware. The beam position can be tracked through an indicator thatreflects the greater of either the X or Y position of the beam. Therecan be a sliding threshold depicted on that indicator that triggersautomatic unwinding when automatic unwind is enabled and themagnification is low enough.

In one embodiment, the drift correction process may include thefollowing steps. After the median blur, the process applies digitalregistration to the live image. The digital registration is applied toeach frame in the drift corrected image sequence, but the softwaresimultaneously saves the raw, unaltered, images into a separate folderthat is viewable in the live view when toggled in the lower indicator.There are no image skips in the raw images or drift correction imagespresented and saved when only a digital registration is applied. Whenthe digital registration hits a percentage, threshold which can be fixedor set by the “Adjustment Threshold” setting, the system then triggers aphysical move. There are applications where a larger or smaller“Adjustment Threshold” setting is preferred. A larger setting may givemore allowable digital registration with fewer physical moves and imageskips. A smaller setting may move more often with less digitalregistration, resulting in a sample that stays more centered in thenative imaging application as well as AXON. This can be preferred whenworking with EELs, EDS or other analytical techniques When a physicalmove is triggered, AXON looks at the “microscope profile” to determinewhich positioner to use depending on the magnitude of the move andresolution of the positioners. AXON may always default to the coarsestavailable positioner if the resolution of the positioner is less thanthe required movement. If the required move is 20 nm and the stage'sresolution is 25 nm then it may default to the next fine positioner, thebeam. However, if the required move is 30 nm, then the stage may be thetriggered positioner. If the stage is the default positioner, thecontrol system may automatically unwind the beam back to 0,0. Thedirection of the physical move is determined by the matrix alignmentfrom the coordinate transform calibrations. The magnitude of move isreliant on the camera or detector calibration by the TEM serviceengineers using common techniques such as MAG*I*CAL.

In terms of drift corrected image sequence, when a physical move istriggered the next image is skipped in the live view and it is not savedto the drift corrected image sequence. It is saved to the raw imagessequence; all images are always saved in raw images. The control systemalso looks to the minimum move time and move pace from the “microscopeprofile” to determine if additional images need to be skipped in casethe image update rate is less than the time it takes to move therequired positioner. Skipping the images while the positioner isphysically moving the sample prevents torn or shadowed images factoringinto drift correction registrations and makes scrubbing through acorrected image sequence more manageable. All images are always saved in“raw images” and the user can always toggle between these two views forthe same time sequence in the live view and AXON Notebook. The driftcorrection process continues through user interruption on the TEM. Thesoftware listens for updates to the TEM column, cameras, and detectorsto determine when to grab a new template to register the image against.

The AXON system can automatically grab a new template and continue thedrift correction process when the following events occur: change inmagnification; change in image physical size; change in pixel area;change binning; change in acquisition time; dwell time; exposure time orintegration time; gain correction enabled; bias correction enabled;change in alpha tilt; beam; stage; change in beta tilt; Beam; stage(only readable if controlled by column like with fusion select); changein brightness; change in contrast; change in convergence angle; changein Z stage; change in defocus; change in region of interest size withinAXON.

The AXON system can pause drift correction and wait until an updatedstate before automatically resuming drift correction when the followingevents occur: beam blanked; column valves closed; and, screen down. Thecontrol system can stop drift correction all together in order to not“fight” the user when the following events occur: stage X/Y movement;beam X/Y movement. Additionally, drift correction may halt the processif the correlation match of the FOV against the template exceeds the“Correlation Failure Threshold”. It may also halt the process if thedigital registration impedes on the region of interest. The driftcorrection registration can accommodate dynamic samples. This isadvantageous for in-situ samples, but even “static” sample change as thebeam interacts with the material or the zone axis changes. A runningfilter may be applied to the original template, morphing it more likethe current image. The aggressiveness of this filter can be fixed or setby the “Template Morphing Factor” setting. A higher setting may resultin a registration template that is more like the current image. Doingthis may slowly move the region of interest in the drift direction, butthis may be necessary to accommodate changing samples. On images that donot change much, it may be advantageous to keep the template morphingfactor low to keep the regions of interest consistent. There are manyways the template morphing setting can be visualized referencing howdynamic a sample is. This can be a variable, slider, fixed settings, orany other type of indicator.

Drift correction can perform a correlation match of the region ofinterest against every pixel array of that size across the image wherethe template is the morphed template. The registration then digitallycenters the region with the highest correlation score in the region ofinterest box. The region of interest can be bounded by a shape overlayon the image in the software. The AXON system does include the option toturn on “Background Assist” through the settings. “Background Assist”continues to prioritize the region of interest, but also manages otherindependent regions of interest to determine overall direction.

In terms of drift control specifications, AXON can correct in X, Y and Zwhen the imaging conditions are appropriate for the expected drift rate.When using proprietary systems, “Experiment Prioritization” mayautomatically help set appropriate ramp rates for the current imagingconditions. However, if the drift is not caused by the proprietaryheating E-chip, the imaging conditions may need to be adjusted. If thecontrol system is not able to keep up with the apparent drift, it canundertake the following actions: reducing the magnification or imagesize; and, speeding up the image acquisition rate.

Focus Assist is a process triggerable from the left bar of the screendisplay of AXON when drift correction is active. The focus region ofinterest is bound by a shape overlaid on the live view. This region ofinterest is moveable within the drift correction region of interest andresizable within limits. Focus assist may not run unless driftcorrection is active to guarantee that the same region of interest isanalyzed in comparative scoring. The primary tools for this process area focus quality score and the defocus adjustment of the microscope.Stage movements are needed during unwinding events but are notautomatically engaged for larger movements due to the unreliable natureof the Z stage positioner on most microscopes. Piezoelectric controlcould also be supported on compatible microscopes.

Focus quality score may be applied to each image, with no history ofprevious scores. This score is reported in the lower indicator as both anumerical score and as a relative quotient. While there are defaultscoring metrics, users can also choose between the below scoring metricsthrough the Focus Assist setting “Focus Score Algorithm”. Each algorithmhas benefits for specific imaging conditions and samples. Variancecalculates the variance of the image by taking the sum of the squareddifferences from the mean after applying an image filter. Inversevariance is calculated as a large value/Variance, which is used forinverted profiles where a decreased variance is preferred. Norm variancetakes the variance and divides by the mean pixel intensity, normalizingfor changes in overall intensity. Inverse norm variance is calculated asa large value/Norm Variance, which is used for inverted profiles where adecreased norm variance is preferred. Norm variance 2 takes the varianceand divides by the mean pixel intensity putting heavier emphasis onnormalizing for changes in overall intensity, better handling groups ofsaturated pixels. Inverse norm variance 2 is calculated as a largevalue/Norm Variance 2, which used for inverted profiles where decreasednorm variance 2 is preferred. Gradient calculates the gradient of theimage by taking the square root of the sum of squares of the gradientmatrix derived from the image after applying an image filter. Inversegradient is calculated as a large value/Gradient, which is used forinverted profiles where decreased gradient is preferred. Gradient 2applies a second filter to the gradient score to enhance edges anddecrease background impact. Inverse Gradient 2 is calculated as a largevalue/Gradient 2, which is used for inverted profiles where decreasedgradient 2 is preferred. Laplacian is based on the square root of thesum of squares of the Laplacian matrix derived from the image. InverseLaplacian is calculated as a large value/Laplacian, which is used forinverted profiles where decreased Laplacian scores are preferred. MaxLaplacian is the maximum of blurred Laplacian matrix. Inverse MaxLaplacian is calculated as a large value/Max Laplacian, which used forinverted profiles where decreased Max Laplacian scores are preferred.Additional scoring metrics can be derived from CTF analysis of an FFT.

A focus quality score is applied to each image, with no history ofprevious scores. Focus quotient provides the history by dividing thecurrent score by the recorded best-ever score. The focus quotient isused for indicating relative focus quality in the lower indicator barand for determining the magnitude of required move. This tells the userand the software how good the focus is compared to its best possiblefocus quality. The history of this focus quotient is reset on each driftcorrection template update so that it accounts for any user interactionon the TEM. There are many reasons as to why a best possible focus scorecan change including reduction in contrast due to carbon contamination.This is worsened in STEM mode with higher dwell times; morphologychanges as the sample reacts to in-situ stimulus or beam; and,morphology changes as the relative axis of the sample rotates. Toaccount for these cases, a filter is applied to the focus quotientmorphing the focus quotient to the current image. The aggressiveness ofthis filter can be fixed or can be set by the setting, “Focus ScoreMorphing Factor”. Whenever the focus quotient is greater than thebest-possible focus score, the score resets to 1. The AXON systemdetermines that an image is in best-possible focus when the focusquotient is 1. As it approaches 0, the image is more and more out offocus, regardless of over or under. When focus assist is initiated, thefocus quotient starts at 1 and it returns to 1 anytime a new template iscreated or anytime the measured focus quality score is above the morphedbest possible. These values can be scaled or interpolated.

In terms of defocus adjustments, while Focus Assist is active, AXONmakes a defocus adjustment on either; every other image or the imageafter the minimum move time, whichever is longer. This ensures thatimages are not mid focus adjustment when sampled for direction andmagnitude of response. The direction of move can be determined by afuzzy logic table where AXON analyzes direction confidence andprobability that the focus is worse. When the direction confidence islow and the focus quotient reduces, the process may reverse direction.When the focus quotient increases, the process may continue in thatdirection. When the confidence is high that direction is correct, theprocess is more resilient to focus quality score reductions to preventreversals when the sample outpaces the controller.

The magnitude of defocus adjustment is determined from the focusquotient and the focus calibration, regardless of direction. As thefocus quotient decreases, the size of response increases. High focusquotients result in small defocus adjustments, small enough that theuser cannot perceive the change, but the sampling statistics maycontinue to improve focus quality. The focus calibration provides thereference for the control system to judge the needed defocus responsefor a given focus quotient.

Z (focus) corrections may always default to the beam (defocus) and notautomatically move the stage or piezo controls. This is because the Zstage may be very unreliable, noisy and has varying hysteresis. Thecontrol system can unwind the beam, much like the X/Y unwind. It can beautomatically triggered through a sliding threshold on an indicator andit can be manually triggered through the unwind button. When the Zunwind is triggered, the control system may step the stage in thedirection of the beam position and then re-focus the sample. Thisprocess continues until the beam position is less than the resolution ofthe Z stage. Each step is determined by the Z stage resolution in themicroscope profile. These moves can be setup so that the beam and stageor beam and piezo are moved in opposite directions in a single move.This process can also be used for unwinding a piezo against the stage.

Experiment prioritization can include ramp-rate control initiated fromAXON to a compatible proprietary Clarity software or any other in-situsoftware, where the Clarity software is still run independently outsideof AXON. As noted earlier, in one embodiment, the control system asdescribed herein can include a software suite provided by tradenameClarity (hereinafter referred to as “Clarity software”, “Clarity”,“control system” or simply “system”). Session types are available thein-situ software products compatible. These session types initiate a2-way connection between AXON and the corresponding in-situ softwarewhich synchs metadata to AXON and AXON sends recommended ramp rates,start, stop, pause, and resume commands to the in-situ software. AXONcan communicate maximum ramp rates within the in-situ softwareapplication that can boost chance of a stable region of interest, ingood focus through temperature changes and to automatically initiatepause/resumes. AXON calculates a recommended ramp rate on connection tothe TEM imaging session and updates anytime the conditions change,regardless if drift correction or focus assist are active. AXON updatesthis ramp rate during drift correction and focus assist to optimizeperformance.

AXON can automatically pause and resume thermal ramps to preventunstable conditions anytime: the focus quality goes below a thresholdwhile focus assist is active—(a) the ramp can pause anytime the focusquotient drops below a fixed value or the setting, “Pause ExperimentThreshold”; or (b) the ramp can automatically resume when the focusquotient is corrected above a fixed value or the setting, “ResumeExperiment Threshold”; the digital registration exceeds a thresholdwhile drift correction is active—(a) the ramp can pause anytime thedigital registration exceeds a fixed value or the setting, “PauseExperiment Threshold”; or (b) the ramp can automatically resume when thedigital registration drops below a fixed value or the setting, “ResumeExperiment Threshold”; anytime the beam is unwinding in X/Y; and,anytime the beam is unwinding in Z.

Anytime the control system triggers an automatic pause, the clarityapplication can alert the user within the Clarity application with textnext to the recommended ramp rate stating, “Held by AXON”. This behaviorcan be configured so that instead of pause and resume commands, agradually decreasing ramp rate and the pause/resume is preferred. The2-way connection triggers UI elements in AXON and in the correspondingClarity product.

In AXON, the following options are provided: “Start Experiment”, “StopExperiment”, “Hold Experiment” and “Resume Experiment”. Additionally,the full workflow of in-situ software such as Fusion Select, PoseidonSelect and Atmosphere 210 can be brought into the AXON user interface. Aconnection indicator in the lower right-hand corner of the indicator bardetailing—product icons; product name; connection status; play button tostart experiment (or apply target); pause/resume button to pause orresume a ramp; stop button to stop the experiment safely cutting powerto the sample or sample support; and current experiment state—active,inactive, automation hold, user hold. (3) Additional notifications onconnection and running state. (4) Default overlay on the live viewdepending on session type.

In the in-situ software, the following options can be provided: (1) Aconnection status—labeled AXON, reporting connection state. (2) AXONRecommended Ramp Rate text and calculated value labeled directly belowthe ramp rate when running a Thermal experiment from Channel A. (3) Textalerting the user when an automation hold is applied right next to therecommended ramp rate.

Regarding connection with the microscope service, AXON computes amaximum correctable drift rate in um/ms from the field of view size,adjustment threshold setting, acquisition time and minimum move time.This allows for enough information to make the needed focus adjustmentsand insures stability in the X/Y correction. A power read from thesample or sample support can allow for more aggressive ramps at lowertemperatures, slowing down over the largest dT/dP sections. The E-chipcan also be used to delineate different behavior when new chips areintroduced.

AXON Synchronicity manages a few data streams all synced throughcorresponding metadata appended through multiple steps in the processes.The images in the session buffer are saved with metadata stemming from:Native imaging OS (for example, TIA or Gatan); Column OS (for example,TFS or JEOL) (TFS or JEOL); and, In-situ system (for example,Protochips). The images are organized in the image buffer between a fewfolders, all saved with the relevant metadata. These images can beexported from the temporary buffer to a permanent folder—again savedwith their metadata but also then exported with a .csv log file of allmetadata appended through each step in the process. The metadata canstart with the image monitoring process in the imaging service. Theimage monitoring process can grab each unique image as a bitmap andattach the relevant metadata from the native imaging OS. Then themicroscope service appends the bitmap metadata with all relevantparameters and sends the package to AXON through the RESTful service.That bitmap is converted to a lossless PNG and the metadata data ismerged with any relevant in-situ metadata. That lossless PNG is savedunedited to the “Raw Images” folder in the session buffer. If the driftcorrection process is running, that image is also saved with allmetadata to the “Drift Corrected” folder in the session buffer after thedigital registration process. If the image was flagged as a singleacquisition rather than a continuous imaging stream, the raw image isagain saved to the “Single Acquire” folder in the session buffer.

The AXON session buffer can be set to operate on a first-in, first-outpriority from the AXON Public Documents directory. The control systemcreates a folder for each session, separating the “Drift Corrected”,“Raw”, “Templates” and “Single Acquires” per session. As the buffer sizeapproaches its maximum limit the earliest images are removed to makeroom for the newest images. These session folders persist for as long asthere are images from that session in the buffer so previous sessionscan still be accessed even if they are not permanently exported if theactive session does not exceed the buffer limit. The images can bemanually moved from this folder or exported using the AXON Notebook andeach image is saved with all relevant metadata, however accessing thismetadata is only possible through the AXON Notebook until the images areexported and the CSV file is created. The AXON Notebook references thisfile structure and requires this organization for easy navigation in theapplication. All images are saved to the buffer at full resolution asacquired from the native imaging software but can be binned ifpreferred. All images exported from the image buffer to permanent diskare saved at full resolution. The user can turn on/off saving each typeof image sequences to maximize the buffer to their preference. The imagebuffer can cover a ranging period depending on the image acquisitionrate and the image saving options presented. If the image update rate is100 ms and both raw images and drift corrected images are enabled forsaving, the image buffer can be as small. However, if the image updateis longer, the image buffer can span a much longer time frame. Thecontrol system can further partition the AXON server hard drive toreserve a block of hard drive for the image buffer and tie the imagebuffer size to available memory rather than a fixed number of images orfixed length of time.

The system has “Data Overlays” and “Image Metadata”. “Data Overlays”enable a layer of text on the live view image updating with each uniqueimage in the live view. Any overlay applied to a session persists intothe AXON Notebook and persists for that session type across multiplesessions. The overlay options are managed through a property grid tablewith the following columns:

The overlay options can include, but are not limited to, the following:

Title: Base Units: AXON: ClarityControlDateTime — date/time ScaleBar —mm/um/nm MicroscopeDateTime — date/time DRIFT CORRECTION:CoordinatedDriftRate Drift Rate: um/ms MatchCorrelation Match: FOCUSASSIST: FocusRoiMean Mean Int: FocusRoi Variance Focus Var: FocusScoreFocus S: FocusQuotient Focus Q: MICROSCOPE: MicroscopeName —MicroscopeType — MicroscopeImagingMode — ConvergenceAngle Conv: radiansSTEMRotation Rotation: deg ImagerMagnificationValue Mag: IMAGE:ImagerName — ImagerImagePhysicalSizeX Size X: umImagerImagePhysicalSizeY Size Y: um ImagerImagePixelsX Size X:ImagerImagePixelsY Size Y: ImagerBinning Binning: ImagerAcquisitionTime— ms ImagerContrast Contrast: ImagerBrightness Brightness: POSITION:CoordinatedPositionX X: um CoordinatedPositionY Y: umCoordinatedPositionZ Z: um CoordinatedPositionA Alpha: degCoordinatedPositionB Beta: deg StageX Stage X: um StageY Stage Y: umStageZ Stage Z: um StageA Alpha: deg StageB Beta: deg BeamX Beam X: umBeamY Beam Y: um BeamZ Defocus: um BeamA Beam Alpha: deg BeamB BeamBeta: deg PixelShiftX Px Shift X: um PixelShiftY Px Shift Y: um IN-SITUATMOSPHERE: HolderTemperature — C HolderPressure — mBar HolderGas —HolderFlowRate Flow Rate: SCCM Tank1Pressure Tank 1: mBar Tank1Gas Tank1: Tank2Pressure Tank 2: mBar Tank2Gas Tank 2: VacuumTankPressure VacTank: mBar VacuumTankGas Vac Tank: HeatingCurrent Holder: mAHeatingResistance Holder: ohms Heating Voltage Holder: mV HeatingPowerHolder: mW ExperimentType — ExperimentLogFile — ExperimentElapsedTime —IN-SITU FUSION: ChannelATemperature — C ChannelACurrent — mAChannelAResistance — ohms ChannelAVoltage — mV ChannelAPower — mWChannelBCurrent Chan B: mA ChannelBResistance Chan B: ohmsChannelBVoltage Chan B: mV ChannelBPower Chan B: mW ExperimentType —ExperimentLogFile — ExperimentElapsedTime — IN-SITU POSEIDON:ChannelATemperature — C ExperimentType — ExperimentLogFile —ExperimentElapsedTime —

A session review tool by the tradename AXON Notebook can operate as aseparate application with a separate installer. It can also be launchedfrom within the AXON main application and is often used duringexperiments to reference the sample's history and previous morphology.The AXON Notebook is used to view and manage images, and to view andmanage metadata from both the microscope and the supported in-situsystems. Data can be exported from the AXON computer and viewedmanipulated elsewhere.

The UI of the AXON Notebook efficiently manages high resolution imagesso that they can quickly be scrubbed, sorted, and manipulated. The UI isdominated by an active image with overlay options and metadataassociated with that image positioned in accordion headers to the right.Underneath the image are some key functions including: Navigation Bar:Time sequenced scrubber with slider that can be dragged to specificimages. On clicking on the bar, the image can be sequenced througharrows on the keyboard or by dragging the slider—(1) First image: jumpto the first image in the session; Previous image: move to the previousimage as shown; Next image: move to the next image as shown; Last image:jump to the last image in the session. (2) Open: Open previous sessionsin the buffer or any session exported to disk. (3) Sync: Refresh thedirectory if an active session is still saving images. (4) Toggle View:Toggle between “Raw”, “Drift Corrected”, “Single Acquire” and “Template”for the same time the active image. At any moment of time, one can viewall other images saved to the closest timestamp. (5) Image Name: Imagename or reference. Save: Permanently export images and metadata to disk.This opens separate window for managing the image export as there areexport options. All available image layers in the main application areavailable in the AXON Notebook as well as all live metadata.

The AXON Notebook can view the active session and previous session thatare still in the buffer or permanently exported to disk.

On clicking save from the AXON Notebook, the software can give exportoptions and status. From the export images window users can set thedestination folder and can export images off the AXON Core server. Anexternal hard drive linked by USB or ethernet network or a cloud drivecan be used for permanent storage of files. Then the user can selectwhich images to export and whether to export with and without overlays.There is an “Export” button to finalize the export and a status barshowing progress. If any errors arise, the notifications can alert theuser and a trace file is automatically created. This process can be runin the background while an image session is still running, and thewindow can be closed and can continue to run.

AXON Synchronicity and all Clarity products can be set up as separateapplications that communicate together. The architecture is set to embedthe workflows of Fusion Select, Poseidon Select and Atmosphere 210 intothe accordion workflow in AXON. Embedding workflows is accomplishedthrough the implementation of a “skinny UI”. The Clarity architecturecan be simplified into passive reporting elements and a workflow. Theworkflow UI is product specific and calls all the controls for theapplication. The reporting element visually depict the data in chars,status panes, notifications, and gas flow diagrams. All UI workflows andreporting elements are separate between native applications and updatesto one application does not ripple into others. Controls are alsoseparate, work on one product does not ripple into the othersautomatically. Embedding workflows without doubling maintenance requiresrestructuring the product specific software so that the workflow ispulled from a new “skinny UI”. AXON would also reference this “skinnyUI”. The user could then run either the native product specificapplication or the workflow within AXON with no changes to workflow.

Some exemplary focal algorithms provided in various implementationsinclude the following. Focus Quality Score: This quality score isapplied to each image, with no history of previous scores. This score isreported in the lower indicator as both a numerical score and as arelative quotient. While there are default scoring metrics, users canalso choose between the below scoring metrics through the Focus Assistsetting “Focus Score Algorithm”. Each algorithm has benefits forspecific imaging conditions and samples: Default: STEM Mode: NormVariance 2; and, TEM Mode: Inverse Gradient 2; Variance: Calculates thevariance of the image by taking the sum of the squared differences fromthe mean after applying an image filter; Inverse Variance: A largenumber/Variance used for inverted profiles where a decreased variance ispreferred; Norm Variance: Takes the variance and divides by the meanpixel intensity, normalizing for changes in overall intensity; InverseNorm Variance: A large number/Norm Variance used for inverted profileswhere a decreased norm variance is preferred; Norm Variance 2: Takes thevariance and divides by the mean pixel intensity squared. Puts heavieremphasis on normalizing for changes in overall intensity, betterhandling groups of saturated pixels; Inverse Norm Variance 2: A largenumber/Norm Variance 2, used for inverted profiles where decreased normvariance 2 is preferred; Gradient: Calculates the gradient of the imageby taking the square root of the sum of squares of the gradient matrixderived from the image after applying an image filter; Inverse Gradient:A large number/gradient used for inverted profiles where decreasedgradient is preferred; Gradient 2: Applies a second filter to thegradient score to enhance edges and decrease background impact; InverseGradient 2: A large number/gradient 2, used for inverted profiles wheredecreased gradient 2 is preferred; Laplacian: Laplacian is based on thesquare root of the sum of squares of the Laplacian matrix derived fromthe image; Inverse Laplacian: A large number/Laplacian used for invertedprofiles where decreased Laplacian scores are preferred; Max Laplacian:Max of blurred Laplacian matrix; and, Inverse Max Laplacian: A largenumber/Max Laplacian used for inverted profiles where decreased MaxLaplacian scores are preferred.

The control system can further provide for the normalization of thescale of these focus scores to make them more easily interpreted acrossdifferent sample areas and magnifications. The control system can alsooperate to estimate the refocus points against the normalized scale. Thecontrol system can generate an autofocus or a refocus routine based oncalibrations at each magnification of focus score and magnitude of Zchange; this can advantageously allow for the focus to be found in asfew moves as possible.

According to various embodiments of the presently disclosed subjectmatter, the control system can operate to keep a sample in focus throughall corrections. The control system can also enable auto-focus of aregion of interest through a visual control tool. The control system canfurther provide for constantly monitoring the focus of a primary regionof interest through the experiment refocusing only when necessary. Toaccomplish this, the control system can operate to keep the samefeatures in the field of view. The control system can provide for thesere-focus points to be tunable via easy indicator handles, editable bythe user, noting the focus envelope. The control system can provide forfocus scores to be normalized and displayed on the graphical userdisplay by the control system as an indicator in a bar shape or in asuitable other shape against an “ideal focus” so that the focus can beeasily manipulated to over or under focus conditions.

In some embodiments, it is advantageous to use continuous adjustment ofdefocus over strategic refocus points. For continuous adjustment ofdefocus, the focus score is normalized by dividing the current focusscore against the best score since the last template. New templates areused anytime the drift correction template is updated because thenormalized focus scores need to be run on the same set of features. Thenormalized score and microscope calibrations set how far the defocus canbe moved. The lower the score, the further the defocus can move;alternatively, the higher the score, the defocus adjustment tends closerto 0. This allows users to manually interact with the algorithm byimproving on the sample and the increasing scores cannot result inmeaningful movements. Any decreasing score gets factored into decisionsto eventually reverse direction. To account for dynamic samples, thefocus scores are morphed through a bump filter, but any other type offilter to bring the best ever score closer to the current score wouldwork. Additionally, the normalized scores are filtered forimage-to-image noise.

According to various embodiments of the presently disclosed subjectmatter, the control system can provide for the Z-axis control to behierarchal using beam, piezo and stage control. Beam control is oftencalled “defocus”. The control system can further automatically pick theright positioner to move depending on the scale needed. The controlsystem can further unwind all smaller movements back to 0 if needed. Forexample, if large movement is needed, the control system can move stageto correct position and zero out the piezo and beam. In one embodiment,an indicator can be used to show the beam position from neutral(preferred) with trigger points to start unwinding the beam back toneutral through stage or piezo moves. We do this in the software todayfor X, Y and Z.

The control system can provide for user specified limits to the“defocus” control so that the beam control does not negatively affectthe image or introduce stigmation. This can also be the case for X, Ybeam control if taken too far from alignment.

In various embodiments, calibrations may be used to improve performanceof the drift correction and to insure accurate moves regardless of theapplication. For example, I some embodiments, the control system can usea sophisticated set of calibrations linking cameras, detectors,positioners, and the in-situ control parameters. The control system canalso constantly monitor performance against these calibrations and couldimprove on the calibrations themselves. In one implementation, acalibration can be setup for each detector at each magnification foreach positioner. These calibrations can help determine rotationaloffset, image timing and magnification focus envelopes. Each positionercan have a calibration where backlash, movement limits, and movementtiming can be quantified. The control system can perform holder specificcalibrations. For example, in one embodiment, the control system createsa “microscope profile” where a connection to the microscope as well asall its associated imaging systems is established. A single microscopecould have different imaging environments and detectors, with each ofthem benefiting from a respective calibration. Each microscope profilecan have a specific set of settings, positioner capabilities, andcompatible imaging systems. The positioner capabilities can include, butare not limited to, the preferred movement range, total available range,resolution, hysteresis, minimum move time and move pace. Each positionercan be characterized—including TEM beam, STEM beam, stage, and piezo.Each positioner can be characterized in the X plane, Y plane, and Zplane and if/when applicable, in terms of alpha (x) tilt or beta (y)tilt as well. These capabilities can be characterized through automatedtest procedures or manual tests with manually entered values. Eachcompatible imaging system may require a specific set of coordinatetransforms that characterizes the rotational offsets and nm/pixel deltasfrom the reported values from the TEM. These calibrations could be savedper imaging system, per detector, per camera, per positioner, and/or permagnification, among others. It is not mandatory to have a calibrationavailable for each of the magnification levels; the control system caninstead be configured or programmed to look for the closest calibratedmagnification of a given positioner on a given imager run through thatimaging system. Focus step size calibrations could be used tocharacterize how far to move the defocus, z stage, or z beam for givenfocus score from best capable or a filtered version of best capable. Thefocus calibrations can be organized per imaging system, per camera, perdetector, per acceleration voltage, and per convergence angle permagnification, among others. It is not required to have a calibration atall magnifications and the control system could look for the closestcalibrated magnification for that convergence angle, or thatacceleration voltage.

The holder specific calibrations can help a user with an imaging origin,X, Y and Z, for a specific holder for easy navigation. Holder specificcalibrations can also contain expected movement models such as, forexample, a drift velocity associated with a temperature change of onedegree Celsius, and ramp rate for heating or cooling holders. In oneembodiment, heating can be combined with any other in-situ parameter,such as heating in gas or liquid. The control system can provide forthese calibrations to be run each session; alternately, the controlsystem can allow for the calibration values to be stored in acalibration database and checked against periodically.

According to various embodiments of the presently disclosed subjectmatter, the control system can automate experiments. The control systemcan also work seamlessly with user interruptions adapting to optimizethe experiment. The control system can constantly measure the field ofview, X, Y position of all positioners, Z position of all positioners,alpha and beta tilt of the holder and image refresh rate to flag anyuser interventions. The control system can then act appropriately towork with the user rather than against the user. For example, in oneembodiment, X/Y drift correction can continue to run when the userchanges the Z position and the focus can still be scored but may notauto-focus while the user is actively changing the Z position. X/Ychanges of any positioner outside of expected vectors can likely meanthat the user is interested in a new region of interest, whereby thecontrol system can proceed to pause or halt drift correction. Imagerefresh rate, commonly a result of the user changing the dwell time inSTEM or exposure time of the camera, may require changes to the in-situstimulus, such as thermal ramp-rate, for example. to better correct fordrift. The control system can provide for such changes to the in-situstimulus. Alpha and beta tilt changes can warrant continued driftcorrection and auto-focus, and the control system can provide for suchcontinued drift correction and auto-focus, as needed.

According to various embodiments of the presently disclosed subjectmatter, the control system can provide for triggering functions for thein-situ stimulus, microscope, camera, or detectors that can be activatedin response to interruptions detected on the microscope. For example,the control system can operate to decrease or pause a thermal ramp ratein-situ stimulus while the user is trying to manually bring the sampleinto focus.

According to various embodiments of the presently disclosed subjectmatter, the control system can provide feedback to attenuate in-situcontrol inputs such as current, temperature and flow rate, preventingthe loss of the primary region of interest. MEMs technology enables veryrapid changes to the sample environment, such as thermal ramps of 1000°C./ms, and these rapid changes could push the sample outside of thefield of view. The max thermal ramp rate achievable while still runningdrift correction can be calculated by the control system from aspectssuch as the active field of view relative to the region of interestsize, positioner timing, image update rate and expected drift rate. Thisattenuation can also be automated by the control system for specificinstances where Z inflections are anticipated due to buckling ofmembranes. Drift correction in the X, Y axis may also be needed toovercome buckling because nanoscale buckling can also move in X, Y, notjust up and down (i.e., not just in Z).

This may not be limited to heating environments. Various in-situstimuluses such as mechanically probing, electrically probing, heating,cooling, pressure changes, or imaging the sample in a fluidicenvironment can enact sudden movements that need attenuation. Thecontrol system can advantageously provide for such attenuations.

According to various embodiments, the control system can furthersimplify the experiment by combining the relevant microscope control andsample stimulus into a single user interface.

It is to be noted that it is not a requirement to bring everything intoa single user interface. Instead, communication methods can be setupbetween applications so that live analysis on the image or microscopeparameter monitoring can issue commands to the in-situ control system.For example, a first application labeled AXON can analyze the liveimages from the microscope and issue pause/resume commands to thein-situ software. Anytime the digital registration exceeds a threshold(a sign that the physical corrections cannot keep up with the drift),the AXON application can issue a pause command to the in-situapplication to pause the stimulus. Then, when the digital registrationfalls below a threshold, the AXON application can send the command toresume. Similarly, when the normalized focus score falls below athreshold (a sign that the sample is going out of focus), the AXONapplication can issue a pause command to the in-situ application,resuming once it rises above a threshold. Instead of issuing pause orresume commands, the AXON application can throttle the ramp-rategradually until the physical corrections can keep up adequately. TheAXON application can also recommend a ramp-rate for certain thermalexperiments. The recommended ramp-rate value can be calculated from themeasured image acquisition rate, field of view size, and some predictivebehavior or characteristic associated with the heating system beingused. The application can update this value according to actual behaviorand the user can just command a target temperature and allow the AXONapplication to completely set and manage the ramp-rate. The controlsystem can also issue pause commands to the in-situ software duringunwinding of the beams or during certain microscope status changes. Thecontrol system can also be configured to stop an experiment depending onpressure changes in the TEM as a safety precaution.

In one embodiment, to help the user enable certain thermal ramp-rates,the control system can operate to show the user how the magnification,active detector size, pixel resolution, binning, dwell rate and exposuretime affect the ability to drift correct. the control system can furtherhelp the user prioritize one or more camera/detector options, microscopesetup, and in-situ stimulus to ensure a stable image within thecapabilities of drift correction, helping the user prioritize certainsettings and then automatically or guiding the user through the setup ofother dependent settings. For example, the user can prioritize a pixelresolution, magnification and thermal ramp rate and the control systemcan operate to automatically pick a dwell rate or exposure time toenable the prioritized settings to keep the image stable and in thefield of view during drift correction. Again, this could be applied bythe control system can to any number of in-situ stimuluses such aspressure changes or any number of microscope parameters.

According to various embodiments of the presently disclosed subjectmatter, in addition to a primary experimental site, the control systemcan operate to use drift vectors to predict the location of a secondaryor even many other imaging sites. Sample movement is often in the samedirection across the active area on heating and cooling holders. Driftvectors applied at one region of interest can be applied by the controlsystem to most of the active area. With beam and holder positioncontrol, the control system can allow for users to easily toggle betweenprimary, secondary, and even tertiary sites during an experiment througha software user interface. These sample locations could be laid out in amap by the control system can for quick control and sites could be keyedas experimental controls to help quantify beam and dose effects on thesample. Sample sites can be a set of X, Y, Z coordinates; alternately,sample sites can be tied to feature recognition of the images.

According to various embodiments of the presently disclosed subjectmatter, to help automate experiments, the control system can developtriggering functions based from several noticed changes to the samplefeatures, microscope conditions, in-situ stimulus source, or in-situstimulus readings. the control system can further enable the user orother software to set triggers to the in-situ function or microscopesettings based on image analysis. For example, the control system candecrease the temperature when a particle size exceeds a certain numberof nanometers. Additionally, the control system can pause a ramp rateand increase camera acquisition rate when the EDS detector picks up ahigher peak of a certain element.

According to various embodiments of the presently disclosed subjectmatter, drift correction of the image enables analysis of a specificfeature, but triggers can be developed by the control system toincorporate multiple sites. For example, when particle size exceeds acertain number of nanometers, a high-resolution acquisition can betriggered by the control system for 2 or 3 predetermined locations—withall sites known to the control system due to the application of driftvectors.

According to various embodiments of the presently disclosed subjectmatter, the control system can also enable users or other software toset triggers to the electron microscope, camera or detector based onin-situ stimulus source or in-situ stimulus readings. For example, theacquisition rate of the camera could be sped up when the measuredresistance of the sample exceeds a certain number of ohms. Additionally,certain detectors could be turned on or off by the control system whenthe sample temperature exceeds a specific temperature. An EELS or EDSmeasurement could be automatically triggered for a specific feature whenthe temperature of the sample reaches a predetermined temperature, andit can automatically turn off to protect the detector once thetemperature exceeds that predetermined temperature. For example, invarious embodiments, the control system can operate the trigger functionin-situations including, for example, decreasing temperature when aparticle speed exceeds a predetermined value; control temperature, ramprate, gas environment, and a similar other attribute falls outside of apredetermined range of values; when particle size, number of particles,electron diffraction, image FFT, and similar other attribute fallsoutside of a predetermined range of values. The control system can alsospeed up acquisition rate when resistance of the sample exceeds apredefined value

The control system can enable users to set triggers based on otherattached equipment such as mass spectrometry, gas, or liquidchromatography, etc. The control system can set a trigger to cause anaction such as adjustment the environment or temperature or taking anEELS measurement once the measured water concentration leaving thein-situ holder is less than 5 ppm, for example. This can advantageouslyremove the guesswork in many existing workflows and help the userautomatically take the next step based on quantitative information. Thetriggers can be programmed through a software program such as Pythonscripting or other specific APIs or a full-blown software developmentkit.

According to various embodiments of the presently disclosed subjectmatter, the control system can provide many interfaces to help users orsoftware develop these triggers. The control system can allow forexperiments to be built in an in-UI (user interface) experiment builder,a visual programming language, a python or other easily accessedprogramming language or through specific APIs or a software developmentkit.

According to various embodiments of the presently disclosed subjectmatter, drift vectors applied by the control system to coordinatemeasurements can help realistically track any number of microscopeparameters over time. The control system can combine measurements ofreal-time dose rate applied to a sample as a function of position on thesample and time, and logging of the cumulative dose (dose ratemultiplied by time throughout the course of an imaging session) appliedto the sample as a function of position. Dose rate can be calculated bythe control system from the electron beam current divided by its area(beam diameter). Dose rate can alternately be measured directly bycommunicating with a faraday cup, a camera and/or a TEM directly. Thesebeam parameters can be tracked by the control system for specificfeatures or for the entire imaged area which may move due to microscopeconditions, natural sample drift, and/or the in-situ stimulus.

Because beam damage is not always obvious from the image, the controlsystem can provide for a method to display where the user has observedthe sample and the amount of dose or dose rate imparted on the sample.According to various embodiments of the presently disclosed subjectmatter, the cumulative dose could, for example, be displayed graphicallyby the control system along with the sample image in the form of aheatmap that would adjust automatically as the sample position andmagnification changes. This would indicate portions of the sample thathad received relatively high dose vs. portions that received lowerdoses. Drift correction could also be applied to this heat map. Further,every X, Y coordinate can be registered according to drift vectors sothat the measured dose rate or cumulative dose is tracked accurately forwhat is happening to each feature on the sample; otherwise, as itdrifts, the measured coordinates can be for the wrong area. Further,maximum dose rate can be tracked by the control system can for that samearea. A total cumulative dose applied can also be tracked by the controlsystem.

According to one or more embodiments, the control system can furthergenerate an automated report based on the registered movement, theapplied in-situ stimulus, and/or the measured microscope parameters.According to one or more embodiments, the control system can allow auser to set an electron dose rate limit or cumulative dose for thesample under observation. The control system can further monitor that anelectron dose rate does not exceed the electron dose rate limit.

The control system is further configured to calculate in real-time anelectron dose rate as a function of a position of an electron microscopelens and time. The control system can use a chip or specific sample tomeasure the current generated by the beam at the sample location forimproving on the reported dose and dose rate. This could represent oneof the calibrations used by the control system.

Cumulative dose for a region of interest can be shown by the controlsystem on the image to show the impact of dose on the sample as afunction of time for beam-sensitive samples. Drift vectors can helpregister this heat map with the movement of specific features. Thiscolored overlay of the field of view developed by the control system caninstruct the user as to what parts of the sample have been exposed to aparticular dose of radiation. With this information, a user candetermine if the user needs to move to a different location or if thesample area is safe to continue imaging with the electron beam.

According to various embodiments, reports could be automated or built bythe user to compare multiple sites for a given in-situ control or as afunction of time. These reporting and graphical techniques provided bythe control system can be used for beam conditions such as dose and doserate; they can be also used for any microscope parameter measured by thesoftware as well as in-situ measurement or stimulus.

According to various embodiments, the control system can also allow auser to set dose rate limits for a sample such that the dose rate cannotexceed a specified threshold value regardless of user inputs that cancontrol the dose rate (beam current, beam size, magnification, andsimilar other parameters.) If a user changes any parameter that wouldcause the dose rate to exceed the threshold value, whether intentionallyor not, the control system can operate to prevent or warn the user fromexceeding the threshold value by limiting the parameters. This wouldadvantageously allow the user to avoid excessive dose rates that canirreversibly damage the sample. These limits to protect the sample canbe applied to other detectors, microscope parameters or the in-situstimulus. Other mechanisms such as colors, counters, or on-screenindicators too can help the user keep track of the total accumulateddose and dose rates, both live and from the image metadata. A dose ratelimit, or a dose budget, can be used in tomography applications to guidea user to only take a certain number of images given the instrumentparameters and to ensure that the total dose to the sample remains underthe dose budget.

According to various embodiments, by measuring and controlling the doseand dose rate, the control system can provide a user with the ability toquickly and quantifiably measure the impact of beam effects on sampleshape, composition, density, electrical characteristics, etc. Userscould quickly measure several reference sites with different doses/doserates to quickly determine benchmark thresholds for these parameters,then image another site with dose/dose rate limits in place to ensurethat beam damage is minimized under known-good conditions. A low-dosereference can be established by the control system to compare againstsites that undergo more extensive or longer imaging With multiple samplesites, these references can be applied to other measured microscopeparameters by the software or for other in-situ stimuluses. In addition,a matrix of conditions can be defined that adjust sample dose and doserate. A thumbnail view can be presented to the user to evaluate visuallywhere sample changes began occurring due to dose. FFTs and othercalculations could be performed on each thumbnail to help identifysample changes as an effect of dose, and cross-correlation can beperformed with a low-dose baseline and the amount of change scored ortagged for interpretation by the user.

Embodiments can further provide for drift correction that combines auser specified region of interest (ROI), background drift and predictivebehavior to track features in the electron microscope then commandspositioners in the electron microscope to center and/or focus the ROI.Embodiments can further provide for predictive behavior that can includeon-the-fly learning of the unique X,Y and Z movement of the specificE-chip and holder combination and applying this knowledge to determinewhere a sample might drift to. Embodiments can further provide fortracking pixel shifts over time to build drift velocity and accelerationvectors. Combining the expected behavior of in-situ holders to improveon those vectors. Embodiments can further provide for allowing the userto draw a region of interest and then commanding the microscope tocenter that ROI in the field of view. Alternatively having a pre-drawnROI and allowing a user to command new center positions which move thesample or beam.

Embodiments can further provide for supporting multiple ROI on a singleimage stream. Embodiments can further provide for supporting a centeringmotion that is not actually the center of the FOV (field of view).Embodiments can further provide for using drift vectors or backgrounddrift and a reference template to determine a sample event for use as aninternal or external flag. Embodiments can further provide for savingimages to file or flagging key data sets. Embodiments can furtherprovide for pause or slow in-situ stimuluses. Embodiments can furtherprovide for updating the actual or indicated region of interest.

Embodiments can further provide for a hierarchal control of positioners.Automatically picking the correct positioner from either the stage,piezo, or beam depending on the size of the needed movement as well asthe amount of movement left before preferable or hard limits.Embodiments can further automatically zero the finer positioner whenmoving coarser positioners. For example, when moving the mechanicalstage, the piezo and beam deflectors can be set to zero and the totalmagnitude of the movement corrected with the mechanical stage. Movingthe beam away from a neutral position can negatively impact the imaging.Accordingly, the control system can include indicators to bringattention to the beam position for X, Y, and Z. The user can set up thecontrol system for “Automatic Unwinding” which can unwind the beamanytime it hits a trigger point on the indicator. Unwinding the beamforcefully moves the next coarser positioner and beam in oppositedirections until the beam is neutralized—without the user losing theregion of interest.

Embodiments can further provide for a user-set or automatic limits ofbeam position, including “defocus”, to prevent unwanted stigmation.Embodiments can also provide for applying a digital correction on top ofa physical correction and saving both image sets to file. Embodiments ofthe presently disclosed subject matter can additionally provide forsaving raw images to file and saving consecutive images as movies, bothcorrected and not corrected.

The data review tool can provide further functionalities when the imagesare all physically and digitally corrected. The control system providesfor a physically and digitally aligned image sequence to enable math andanalysis applied to a single frame to be extrapolated across an entireimage sequence. The control system can further provide for supportingstatistics on a single sample site over time, plotting any metadata orderivations, intensity analysis, FFTs, and similar other statisticsacross multiple images to thereby provide for the ability to build ahistory of the analysis.

Focus control can further allow for continuous defocus adjustmentsscaled by a normalized focal score. The control system can allow fornormalizing the focal score, morphing the normalization for changingsamples and filter out noise. The control system can further allow forcontinuous defocus adjustments to be run along with user adjustments.

The effectiveness of the control system is further enhanced by theprovision of tunable filters to morph the original registration templateinto the current live view, and the ability to completely reset thistemplate strategically when a user changes FOV, imaging conditions, orsimilar other key items on the microscope.

The control system manages the image buffer across multiple sessionswith files written to disk rather than held in memory. The controlsystem further provides for scheduled or continuous cleanup of thebuffer and further provides for the ability to export images from thesession directory to other permanent drives. Some or all of these imagescan be held at a priority overriding the buffer cleanup. Users can tagimages to override first-in-first-out buffer rules with processes tomake sure that the rest of the buffer can still be managed withoutoverwhelming the hard-drive space. The control system further includesindicators to show the size of the buffer and the size of theprioritized images. The control system can further operate to reduceoverall data size of the buffer when running out of storage size. Inorder to reduce storage space, the control system operates to save onlythe changing pixels rather than entire image per frame and stitch themtogether in the live view; the control system also operates to bin downimages where correlations are too similar, or the control systemoperates to store average pixels when correlation are similar. Thecontrol system also uses fast dwell times over a longer period of timewith physical corrections to build live EDS maps of a sample site. Thecontrol system can further use similar workflows for EELs. The controlsystem may save secondary sites at a lower magnification and may use thesecondary site data to do more than just analyze beam effects. Thecontrol system can provide for automatically jumping between a specificsample site and a much lower magnification view of the sample to put thesample sites into perspective. The control system can also provide forautomatically jumping between a set of sample sites and a much lowermagnification view of the sample to put the sample sites intoperspective. The control system further operates to enabling users inthe AXON Notebook review tool, for example, to scrub through differentsites and their macro view as a function of time to see relativechanges.

The control system can also be configured such that dedicated servicesthat run on specific machines are structured differently so that imageprocessing could be done on the camera or microscope PCs rather thanservices that send images and information to the computing device onwhich the control system is executing.

The control system can save digitally registered and raw imagestogether. The image buffer can be managed across multiple sessions withdata files written to disk rather than held in memory. The controlsystem can further allow for scheduled cleanup or continuous cleanup ofthe image buffer and the ability to export images from the sessiondirectory to other permanent drives.

According to one implementation, some images can be held at a prioritystatus, overriding the buffer cleanup. The system can further provideusers with the ability to tag images to override buffer cleanup based onfirst-in-first-out buffer rules with processes to make sure that therest of the buffer can still be managed without overwhelming thehard-drive space. The system can further include indicators used to showthe size of the buffer and the size of the prioritized images.

Embodiments can further provide for autofocus or refocus routine to findthe ideal focus, normalization scale and refocus points in as few movesas possible. Embodiments can also provide for focus can be found in asfew moves as possible based from a calibration of focus score and Zdistance at each magnification. Embodiments can additionally provide fora visual focus control tool for electron microscopes built from anormalized focus score versus calculated ideal. Embodiments can alsoprovide for user set refocus handles and further for over focus andunder focus. Embodiments can also provide for ability to drag the actualfocus on the normalized scale to easily over and under focus the sample.Embodiments can additionally provide for combining positioner, lens, andholder calibrations with actual behavior to improve direction andmagnitude of commanded movements. Embodiments can further provide formonitoring X/Y position, Z position, alpha/beta tilt, and image refreshrate to flag any user interruptions. Embodiments can further provide formany variations of the decision matrix with the user duringinterruptions vs. against the user. Embodiments of the presentlydisclosed subject matter can further provide for tracking constantbehavior of interruptions to improve on expected models. Embodiments canalso provide for triggering new behavior on the in-situ control,microscope, camera, or detector from interruptions detected on themicroscope. Embodiments can additionally provide for decreasing orpausing a thermal ramp rate when user is trying to manually bring thesample into focus by adjusting the defocus knob. Embodiments can furtherprovide for automatic attenuation of in-situ control inputs such as ramprate to prevent the loss of the primary ROI. Embodiments can provide forautomatic attenuation of in-situ control inputs to overcome knownperformance of the control system such as film buckling at specifictemperatures. Embodiments can further provide for a software algorithmthat can calculate max ramp rate of the stimulus from the active fieldof view relative to ROI size, positioner timing, image update rate andexpected drift rate.

Embodiments can provide for a software tool that can help users set themagnification, active detector size, pixel resolution, binning, dwellrate and/or exposure time to achieve specific thermal ramp rates.Embodiments of the presently disclosed subject matter can furtherprovide for monitoring, controlling, and/or altering pressure changes orany stimulus change that could cause drift. Embodiments can additionallyprovide for allowing the user to prioritize one or more camera/detectoroptions, microscope conditions, and in-situ stimulus to ensure a stableimage within the capabilities of drift correction. Embodiments of thepresently disclosed subject matter can further provide for helping theuser prioritize certain settings and then automating the setup of otherdependent settings. Embodiments can also provide for the user toprioritize a pixel resolution, magnification and thermal ramp rate andthe software would automatically pick a dwell rate or exposure time toenable the prioritized settings to keep the image stable and in the FOV(field of view) during correction. Embodiments can further provide forapplying drift vectors to predict the location of secondary or manyother imaging sites and allowing users to easily toggle between sites.

Embodiments can further provide for an indicator to normalize drift rateand alert the user of when movement is slow enough for a high-resolutionacquisition. Embodiments can allow for EDS or EELS spectral or maps tobe taken of a sample that is moving due to thermal effects or simply thesample reaction itself. Though this method of drift correction thataccounts for sample movement as well as sample changes, EDS maps can berealigned based on the drift corrected STEM data. EDS typically requireslong exposures or the integration of many short exposures of the samesample area in order to accumulate enough signal to build a map orspectrum with sufficient signal to noise. Prior art solutions only allowfor an exact cross correlation and digital realignment of frames thatare moving, but this technique may not work for a sample that is movingtoo quickly, too far or is changing. The approach for drift correctiondescribed in this subject matter can allow for EDS data to be taken atintervals defined by the user, then realigned based on the simultaneousSTEM images taken. Furthermore, the user can decide to integrate framesin order to build a higher signal to noise image stack. This newtechnique would allow for the creation of video clips using EDS mapsthat show the changing composition of a sample through time. The sametechnique could be done using EELS maps assuming a suitable simultaneousTEM image for drift correction could be acquired.

Embodiments can further provide for enabling the user to set triggers tothe in-situ function based on image analysis and subsequently adjust thein-situ environment through control of the in-situ equipment.Embodiments can also provide for decrease temperature when particle sizeexceeds a predetermined size in nanometers. Embodiments can additionallyprovide for controlling any in-situ stimulus based on image analysistechniques of the acquired image through TEM or STEM. Embodiments canfurther provide for controlling temperature and/or ramp rate, gasenvironment, and similar other attributes based on particle size, numberof particles, electron diffraction, image FFT, and similar otherparameters.

Embodiments can provide for controlling any in in-situ stimulus based onother electron microscope column detectors including EDS (EnergyDispersive X-Ray Spectroscopy) and EELS (Electron Energy LossSpectroscopy) and similar other techniques. Embodiments can furtherprovide for controlling temperature and/or ramp rate, gas environment,etc. based on elemental ratio from EDS maps, reduction of a samplethrough EDS (Energy Dispersive X-Ray Spectroscopy) and EELS (ElectronEnergy Loss Spectroscopy) and similar other techniques. Embodiments canfurther provide for enabling the user or other software to set triggersto the electron microscope, camera or detector, other in-situ equipmentbased on in-situ stimulus readings. Embodiments further provide forspeeding up acquisition rate when resistance of the sample exceeds apredetermined resistance value in ohms. Embodiments disclosed herein canfurther provide for pump-purge cycle routine until the total waterconcentration as read by an integrated mass spectrometer reads below apredefined valued, for example, <5 ppm. Embodiments can further providefor interfaces to help researchers build experiments and make customtriggers either through an in-UI (user interface) experiment builder,visual programming language, scripting language, a Python wrapper, a API(application programming interface), and/or a SDK (software developmentkit).

Embodiments can provide for tracking the total accumulated dose andmaximum dose rate of a specific sample site to help users quantify beamdamage of a site. Embodiments can further provide for a sample site tobe a set of coordinates or features in the image tracked by the controlsystem. Embodiments can further provide for a heat map that sums therectangular regions tracked by software to visualize the totalaccumulated dose and maximum dose rate of a wider field of view.Embodiments can also provide for a visualizer to compare beam effectsfor a single site or across multiple sites at specific times or forspecific in-situ stimulus conditions. Embodiments can further providefor a heatmap for sample positions.

Embodiments can provide for an automatic report generator that comparessample sites for a given in-situ control or as a function of time.Embodiments can further provide for limits for dose, dose rate, othermicroscope parameters or in-situ stimulus. Embodiments can additionallyprovide for software tools to help the user avoid excessive stimulus toa region of interest. Embodiments can also provide for a softwareroutine to allow the user to set the maximum total accumulated does ordoes rate and prohibits or warns the user when these limits areapproaching or surpassed in each region. Embodiments can further providefor establishing a reference site to compare against sites that gothrough more rigorous imaging or in-situ environmental changes.

FIG. 1 is a schematic representation of drift correction that combinesuser specified ROI (region of interest), background drift, andpredictive behavior to track features in the electron microscope thencommands positioners in the electron microscope to center and/or focusthe ROI, according to one or more embodiments of the presently disclosedsubject matter. The smart drift correction module is communication witha position control module and an imaging control module. The positioncontrol module is configured to communicate with positioners, andfurther to adjust the setting of the positioners based on instructionsreceived from the smart drift correction module. The imaging controlmodule is configured to communicate with various aspects of imagingincluding acquiring images based on instructions received from the smartdrift correction module.

FIG. 2 is a schematic representation showing the details of reactivedrift correction, according to one or more embodiments of the presentlydisclosed subject matter. The steps of the reactive correction processproceed according to the flow chart illustrated in FIG. 2 according toat least one embodiment of the presently disclosed subject matter.

FIG. 3 is a schematic representation showing on-the-fly learning ofunique X, Y and Z movement of the E-chip and holder in combination ofpredictive behavior of where it may drift to enhance correctionprocesses, according to one or more embodiments of the presentlydisclosed subject matter

FIG. 4 is a schematic representation of software tracking pixel shiftsover time to build drift velocity and acceleration vectors. Combiningthe expected behavior of in-situ holders to improve on those vectors,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 8 is a flow chart wherein a software module that forms part of thecontrol system that uses drift vectors, background drift and/or areference template to determine when a sample is changing, and usingthis information as an internal or external flag, according to one ormore embodiments of the presently disclosed subject matter.

FIG. 9 is a flowchart illustration of a software module that forms partof the control system that is configured to trigger to a camera, adetector, a microscope or in-situ; According to one or more embodimentsof the presently disclosed subject matter, examples of trigger actionsundertaken by this software module include pause or slow in-situstimulus, save off imaging buffer, increase acquisition rate, or moveposition.

FIG. 10 is a flowchart illustrating software module that forms part ofthe control system using a hierarchal control of positioners,automatically picking the correct positioner from either the stage,piezo or beam depending on the size of the needed movement and theamount of movement left before preferable or hard limits, according toone or more embodiments of the presently disclosed subject matter.

FIG. 11 is a graphical illustration of software module that forms partof the control system. As illustrated in FIG. 11, the control system isconfigured for applying a digital correction on top of a physicalcorrection and saving consecutive images as movies, both corrected andnot corrected, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 12 is a flow chart illustrating software module that forms part ofthe control system running an autofocus or refocus routine to find theideal focus, normalization scale and refocus points in as few moves aspossible, according to one or more embodiments of the presentlydisclosed subject matter. FIG. 13 is a flow chart illustrating a focusscoring sweep, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 14 is a graphical representation of a visual focus control tool forelectron microscopes built from a normalized focus score vs. calculatedideal with user set refocus handles and the ability to drag the actualfocus against a normalized scale, over and under focused, according toone or more embodiments of the presently disclosed subject matter.

FIG. 15 is a software module that combines positioner, lens, and holdercalibrations with actual behavior to improve direction and magnitude ofcommanded movements, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 16 is a flowchart of software module that forms part of the controlsystem that monitors X/Y position, Z position, alpha/beta tilt and imagerefresh rate to flag any user interruptions, according to one or moreembodiments of the presently disclosed subject matter. FIG. 17 is aflowchart of software module that forms part of the control system thatmonitors X/Y position, Z position, alpha/beta tilt and image refreshrate to flag any user interruptions but designed to continue thecorrection process to better maintain drift vectors through theinterruption, according to one or more embodiments of the presentlydisclosed subject matter. FIG. 18 is a flowchart of software module thatforms part of the control system that monitors X/Y position, Z position,alpha/beta tilt and image refresh rate to flag a change to an in-situstimulus such as temperature or pressure, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 19 is a diagrammatic representation of software module that formspart of the control system which triggers new behavior on the in-situcontrol, microscope, camera or detector from interruptions detected onthe microscope, according to one or more embodiments of the presentlydisclosed subject matter. FIG. 20 is a diagrammatic representation ofsoftware module that forms part of the control system which takes userinterruptions on the microscope and improves on expected models orprocesses, according to one or more embodiments of the presentlydisclosed subject matter. FIG. 21 is a schematic representation ofsoftware module that forms part of the control system with automaticattenuation of in-situ control inputs such as ramp rate to prevent theloss of the primary ROI, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 22 is a flowchart of software module or algorithm that forms partof the control system that calculates max ramp rate of the stimulus fromthe active field of view relative to ROI size, positioner timing, imageupdate rate and expected drift rate, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 23 is a flowchart of software module that forms part of the controlsystem that helps users set the magnification, active detector size,pixel resolution, binning, dwell rate and/or exposure time to achievespecific thermal ramp rates, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 24 is a schematic graphical representation of software module thatforms part of the control system which allows the user to prioritize oneor more camera/detector options, microscope setup, and in-situ stimulusto ensure a stable image within the capabilities of drift correction,according to one or more embodiments of the presently disclosed subjectmatter. Helping the user prioritize certain settings and then automatingthe setup of other dependent settings.

FIG. 25 is a schematic representation of software module that forms partof the control system which applies drift vectors to predict thelocation of secondary or many other imaging sites and allowing users toeasily toggle between sites, according to one or more embodiments of thepresently disclosed subject matter.

FIG. 26 is a schematic graphical representation of an indicator tonormalize drift rate and alert the user of when movement is slow enoughfor a high-resolution acquisition, according to one or more embodimentsof the presently disclosed subject matter.

FIG. 27 is a diagrammatic representation of software module that formspart of the control system that enables the user or other softwaremodules to set triggers to the in-situ function based from imageanalysis, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 28 is a diagrammatic representation of software module that enablesthe user or other software modules to set triggers to the electronmicroscope, camera or detector based from in-situ stimulus readings,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 29 is a diagrammatic representation of interfaces that helpresearchers build experiments and make custom triggers, according to oneor more embodiments of the presently disclosed subject matter.

FIG. 30 is a schematic representation of software tracking module thetotal dose and dose rate of a specific sample site to help usersquantify beam damage of a site for a specific feature, according to oneor more embodiments of the presently disclosed subject matter.

FIG. 31 is a schematic graphical representation of software visualizermodule to compare beam effects for a single site at specific times orfor specific in-situ stimulus conditions, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 32 is a schematic graphical representation of software visualizermodule to compare beam effects for multiple sites at specific times orfor specific in-situ stimulus conditions, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 33 is a schematic graphical representation of software automaticreport generator module that compares sample sites as a function oftime, according to one or more embodiments of the presently disclosedsubject matter.

FIG. 34 is a schematic graphical representation of software automaticreport generator module that compares sample sites for a given in-situcontrol, according to one or more embodiments of the presently disclosedsubject matter.

FIG. 35 is a schematic representation of software module which limitsdose, dose rate or other microscope parameters or in-situ stimulus,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 36 is a schematic graphical representation of software module whichlimits dose, dose rate or other microscope parameters or in-situstimulus, according to one or more embodiments of the presentlydisclosed subject matter. The software interface establishes a referencesite to compare against sites that go through more rigorous imaging orin-situ environmental changes, according to one or more embodiments ofthe presently disclosed subject matter.

FIG. 37 is a diagrammatic representation of an example for how to trackmultiple sample sites across the entire imageable area for quicknavigation through UI or triggers, according to one or more embodimentsof the presently disclosed subject matter.

FIG. 38 is an illustrative example of one or more regions of interestidentified on the live image feed, according to one or more embodimentsof the presently disclosed subject matter.

FIG. 39 is an illustrative diagram of a basic communication architecturefor the software module that forms part of the control system, accordingto one or more embodiments of the presently disclosed subject matter.

FIG. 40 is diagrammatic representation of a filtering technique toreduce the background noise of an image, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 41 is diagrammatic representation of multiple regions of interestpresented against total field of view, according to one or moreembodiments of the presently disclosed subject matter.

FIG. 42 is diagrammatic representation is an example of reportgeneration from multiple sites for a given time or in-situ stimulus,according to one or more embodiments of the presently disclosed subjectmatter. The metadata can advantageously be of value during and after theexperiment. The control system may permit users to plot metadata andfilter all metadata linked to the images. For example, the controlsystem can allow a user to plot temperature vs. time, and then selectonly those images involved in specific temperature transitions. Asanother example, the control system can allow a user to plot focusquality scores and filter a specific image set for creating timesequences, wherein the specific image set only includes images that arein good focus.

FIG. 43 is diagrammatic representation of a control system in the formof a chart, according to one or more embodiments of the presentlydisclosed subject matter.

FIG. 44 through FIG. 57 illustrate various portions of the controlsystem of FIG. 45 whereas FIG. 58 through FIG. 68 are schematicgraphical representations of a workflow to automate in-situ experiments,according to one or more embodiments of the presently disclosed subjectmatter.

FIG. 58 is a graphical representation of the first step in an automatedexperimental workflow wherein the software module helps users find theoperational area for the experiment which is often a subset of theentire moveable range in X, Y and Z axes. This is the area where samplecan be viewed and where in-situ stimulus can be applied.

FIG. 59 is a graphical representation of the second step in an automatedexperimental workflow wherein the software module helps users tagspecific regions of interest within the operational area. The softwaremodule can save locations and help users manually or programmaticallynavigate to these key areas easily referenced by thumbnails of thesample morphology and a coordinate in X, Y and Z axes of location on amap.

FIG. 60 is a graphical representation of the third step in an automatedexperimental workflow wherein the software module helps users review thetagged regions. This can be an automatic or manual step for users todown select the most important regions.

FIG. 61 is a graphical representation of the fourth step in an automatedexperimental workflow where users load or build an automated experiment.The in-situ stimulus profile can be created. Additionally, imagecaptures at all regions of interest identified earlier can be manuallytriggered or programmed as part of the experiment.

FIG. 62 is a graphical representation of the fifth step in an automatedexperimental workflow where the programmed experiment is physically run.The software module would apply the programmed stimulus and capturechanges at all tagged regions of interest as programmed in theexperiment setup. The sample drift is tracked throughout the experiment.

FIG. 63 is a graphical representation of the 6th step in an automatedexperimental workflow where the user can easily review the changes ofeach tagged region of interest as a function of in-situ stimulus andmicroscope conditions.

FIG. 64 is a graphical representation of an alternative view of the 6thstep in an automated experimental workflow where the user can easilyreview experimental data indexed with the images of a single region ofinterest captured during the automated experiment to visualize how asingle sample site changed over time.

FIG. 65 is a graphical representation of an alternative view of the 6thstep in an automated experimental workflow where the user can easilyreview experimental data indexed with the images captured among multipleregions of interest during the automated experiment to see how multiplesites looked at specific times.

FIG. 66 is a schematic graphical representation showing how taggedregions at multiple sites can be tracked even if only 1 region ofinterest is in the field of view.

FIG. 67 is a schematic graphical representation of an architecture wherethe control software running on a control software CPU utilizes a singlemicroscope service on the microscope CPU. The microscope service canhandle all needed microscope and imaging controls needed by the controlsoftware in this architecture.

FIG. 68 is a schematic graphical representation of an architecture wherethe control software running on the control software CPU utilizes both amicroscope service on the microscope CPU and an imaging service on theimaging CPU. The microscope service can handle all needed microscopecommands and the imaging service handles are imaging commands needed bythe control software in this architecture. The microscope CPU andimaging CPU can be the same CPU or different CPUs in this architecture.

FIG. 69 is a schematic graphical representation of a microscope serviceclass needed for microscope commands and imaging commands Commandsinclude getting images, getting microscope metadata, getting imagingmetadata and setting positioners or imaging conditions dictated by thecapabilities detailed in the control software.

FIG. 70 is a schematic graphical representation of a microscope profile.The microscope profile can be used to detail the network architecture,positioner capabilities and store needed calibrations of the microscopeand imaging system. Calibrations are used to detail positionercapabilities, the rotational offset of positioners against each imagerfor specific imaging conditions and the relationship between positionermoves against focal depth for specific imaging conditions. FIG. 71 is avariation of FIG. 70 where the microscope profile is created fromcontent and capabilities from an imaging service and a microscopeservice rather than a single service.

FIG. 72 is a schematic graphical representation of a high-level processto connect to the microscope and imaging software and transmit uniqueimages with all relevant metadata to the control software. FIG. 73 is aschematic graphical representation of a more detailed image monitoringprocess that can be used to determine unique images from a continuousimage feed and transmit the unique images to the control software. FIG.74 is a schematic graphical representation of a process used to connectto the required services. Services could include microscope services,imaging services and services built to communicate to any number ofdetectors or ancillary equipment involved in the experiment.

FIG. 75 is a schematic graphical representation of a test connectionprocess. On successful connection, a microscope profile can beautomatically created detailing the network configuration and pullingover any specific service settings. FIG. 76 is a schematic graphicalrepresentation of a process to calibrate for the X/Y rotational offsetbetween a positioner and an imager. This process involves moving apositioner in a known direction accounting for calibrated resolution andbacklash of the positioner and calculating the resulting coordinatetransform. FIG. 77 is a schematic graphical representation of a processto handle multiple positioners capable of calibrating under specificimaging conditions. FIG. 78 is a schematic graphical representation of aprocess to calibrate the required Z adjustment needed to correct for animage quality score change under specific imaging conditions.

FIG. 79 is a schematic graphical representation of a process to rundrift correction in X, Y and Z. Where Z focus corrections are continuousadjustments based on a history of focus quality scores of a region ofinterest in an X/Y drift corrected sequence. FIG. 80 is a schematicgraphical representation of a process to start image acquisitionremotely from a control software. FIG. 81 is a schematic graphicalrepresentation of a process to stop image acquisition remotely from acontrol software.

FIG. 82 is a schematic graphical representation of a process to move asample to a specific location in the field of view. This process can beused to manually center a sample in the field of view, it can be used bydrift correction process to automatically center a sample in the fieldof view or it can be used to move any specific region of interest to anylocation within the field of view.

FIG. 83 is a schematic graphical representation of a process todetermine if the image has stabilized after a commanded move by themicroscope. This process can be used to remove frames from calculationsneeded for correction algorithms Additionally, this process can be usedto leave the resulting drift corrected image sequence free of framesblurred by the physical corrections of microscope positioners.

FIG. 84 is a graphical representation of key controls and indicatorsthat could enhance the drift correction experience in the controlsoftware user interface. These indicators can include key metadata aboutthe microscope status, in-situ status and imaging conditions.Additionally, these indicators in the user interface can enable users toswitch between raw images and digitally registered images in the liveview and give insight into the number of images saved into the imagebuffer in the active session—the total number of images and thepercentage of available buffer. The drift rate of the region of interestcan be displayed numerically as a distance over time or as moregraphical indicators. The X and Y beam location can be displayed ascoordinates or as a sliding indicator against preferred range. The Zdefocus location can be displayed as a value or as a sliding indicatoragainst preferred range. Buttons or automated trigger thresholds can becreated to unwind X/Y beam or Z defocus back to 0,0,0 without losing thesample.

FIG. 85 is a graphical representation of key controls that can enableusers to review the history of a session from the software userinterface. An image scrubber can be used to quickly navigate betweenframes. The raw images, drift corrected images and single acquisitionscould be organized by time so that users could easily scrub through adrift corrected sequence and then toggle the display to show thecorresponding raw image or nearest single acquisition.

FIG. 86 is a graphical representation of a method by which users couldtag specific frames and time sequences with a description from thecontrol software user interface. The tag feature could be used to givepriority to images in the buffer so that they overridefirst-in-first-out buffer rules preserving the key frames from beingremoved during automated buffer clean-up processes. Additionally, taggedframes could be highlighted in review tools or metadata plots for easynavigation. Tagged frames could be exported to data drives separatelyfrom the entire session buffer.

FIG. 87 is a graphical representation of key settings that a user couldmanipulate to customize the active image buffer and session management.User settings could be used to state the image buffer location, size,cleanup properties, what images are saved and the percentage of thebuffer that can be allocated to preferred images.

FIG. 88 and FIG. 89 are graphical representations of how the controlsoftware could be used to build a microscope profile characterizing thenetwork configuration, positioner capabilities and required calibrationsneeded by the control software to function appropriately. The controlsoftware could enable raw control of the microscope functions tomanually perform needed calibrations or provide automated processes.FIG. 90 and FIG. 91 are graphical representations of how the controlsoftware could manage calibrations specific to imaging conditions andimagers. FIG. 92 is a graphical representation of a user interfaceenabling users to dictate specific types of in-situ experiments orworkflows that may change the behavior or options of the controlsoftware.

FIG. 93 is a graphical representation of a user interface enabling keyworkflow functions such as connect, drift correct, focus assist, reviewsession, close session, settings and exit. Users can interact with thelive image view with key indicators and controls easily viewable throughthe experiment.

FIG. 94 is a graphical representation of a user interface comprised ofindicators and triggers that enhance the correction experience.Additional user interface options can manipulate or overlay data on thelive image to customize the experience.

FIG. 95 is a graphical representation of a user interface for a sessionreview tool where users can view images and metadata. Sessions could bemoved to permanent storage in many file formats such as image stacks,single frames, videos, or databases from this tool.

FIG. 96 is a graphical representation of user settings that can bemanipulated to customize the experience. FIG. 97 is a graphicalrepresentation of a user interface where focus assist and focus assistcalibrations can be enabled while viewing the live image. FIG. 98 is agraphical representation of how the control software or associateddocumentation could communicate the relationship between imageacquisition rate and field of view as a function of acceptable driftrate.

FIG. 99 is a graphical representation of how a focus algorithm canutilize the focus quality score in STEM mode to drive toward an apexthrough adjustment of defocus. Focus quality is determined by scoringthe contrast of the region of interest. The size of steps is differentdepending on the imaging conditions, including the magnification amongother parameters.

FIG. 100 is a graphical representation of how a focus algorithm canutilize the inverse of the focus quality score in TEM mode to drivetoward an apex through adjustment of defocus. Focus quality isdetermined by scoring the contrast of the region of interest. Theinverse of this scoring technique is required in TEM mode. The size ofsteps is different depending on the imaging conditions, including themagnification among other parameters.

FIG. 101 is a graphical representation of the overall data flow for acontrol service interacting with in-situ systems, an imaging service, amicroscope control service and eventually exporting images and metadatapermanently to disk. FIG. 102 is a graphical representation of a userinterface for prior art in-situ heating software. FIG. 103 is agraphical representation of a user interface where the control softwarerecommends ramp rates and communicates automated pauses/resumes andconnection status within the in-situ software and control software.

FIG. 104 is a graphical representation of a user interface wheremetadata from the in-situ system, microscope, imaging system and anyother connected systems can be viewed and overlaid onto the live displayand session or image review tool. Each image is saved with metadata thatcan be overplayed for users to see how parameters changed on the driftcorrected sequence over time.

FIG. 105 is a graphical representation showing an example of an existingin-situ software suite with unique workflows and reporting elementspushing data to another software that synchronizes data FIG. 105Bdetails an example of a workflow in an existing in-situ software vs thereporting elements in that software.

FIG. 106 is a graphical representation showing how the software suitedescribed in FIG. 105 could have workflows shared between the nativein-situ software and an embedded element within the control software. Inthis architecture, the entire in-situ user interfaces or certain subsetsof in-situ user interfaces can be embedded in the control software userinterface—possibly with a shared codebase. Reporting elements can beadded as image metadata and incorporated into a common metadata plottingtool, log file or database.

FIG. 107 is a graphical representation showing an example of the userinterface of an existing in-situ software and how certain elements ofthat user interface can be embedded into the control software givingusers access to the live image, in-situ control and other features froma single tool. FIGS. 107A and 107B show the user interface of anexisting in-situ software. FIGS. 107C and 107D show how the workflow andreporting elements could be embedded or built in the control softwareuser interface.

FIG. 108 and FIG. 109 are graphical representations of user interfacesused for existing in-situ control software, highlighting the criticalelements that can be embedded into the control software workflow anduser interface.

FIG. 110 through FIG. 115 represent a graphical flow chart detailing aworkflow where the control software can help users effectively quantify,knowingly operate within, and review the effects of cumulative dose ormaximum instantaneous dose rate on an experiment. FIG. 110 is a summaryof an example workflow. FIG. 111 describes 2 methods where the controlsoftware can be used to help calibrate the true dose or dose rate at thesample so that experimental conditions are known and can be replicated.FIG. 112 shows how the control software can help users quantify anddetermine how much cumulative dose or instantaneous dose rate is toomuch for a sample and save the limits as a dose budget. FIG. 113describes how the control software can help track the cumulative dose orinstantaneous dose rate that operate within the established dose budget.FIGS. 114 and 115 describe methods that the control software can use toreview sample sites and further quantify the effects of dose on theirexperiment.

As may be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium (including, but not limitedto, non-transitory computer readable storage media). A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the lattersituation scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be noted,in some alternative implementations, the functions noted in the blockmay occur out of the order noted in the figures. For example, two blocksshown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration but are not intended tobe exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiments were chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

These and other changes can be made to the disclosure in light of theDetailed Description. While the above description describes certainembodiments of the disclosure, and describes the best mode contemplated,no matter how detailed the above appears in text, the teachings can bepracticed in many ways. Details of the system may vary considerably inits implementation details, while still being encompassed by the subjectmatter disclosed herein. As noted above, particular terminology usedwhen describing certain features or aspects of the disclosure should notbe taken to imply that the terminology is being redefined herein to berestricted to any specific characteristics, features, or aspects of thedisclosure with which that terminology is associated. In general, theterms used in the following claims should not be construed to limit thedisclosure to the specific embodiments disclosed in the specification,unless the above Detailed Description section explicitly defines suchterms. Accordingly, the actual scope of the disclosure encompasses notonly the disclosed embodiments, but also all equivalent ways ofpracticing or implementing the disclosure under the claims.

What is claimed is:
 1. A control system configured for sample trackingin an electron microscope environment, the control system comprising: amemory; a processor; and a microscope control component, the controlsystem configured to: register a movement associated with a region ofinterest located within an active area of a sample under observationwith an electron microscope, wherein the registered movement includes atleast one directional constituent, wherein the region of interest ispositioned within a field of view of the electron microscope; direct anadjustment of the microscope control component to one or more of:dynamically center a view through the electron microscope of the regionof interest, and dynamically focus the view through the electronmicroscope of the region of interest; wherein the adjustment comprisesone or more of: a magnitude element, and a direction element.
 2. Thecontrol system of claim 1, wherein the control system is furtherconfigured to apply an in-situ stimulus to the region of interest,wherein the adjustment comprises a drift correction along an x-axis anda y-axis.
 3. The control system of claim 2, wherein the control systemis further configured to apply an in-situ stimulus to the region ofinterest, wherein the adjustment comprises a drift correction along az-axis.
 4. The control system of claim 1, wherein the control system isfurther configured to alert a user when the registered movement is belowa predetermined value or predetermined rate.
 5. The control system ofclaim 1, wherein the microscope control component is in electroniccommunication with one or more of: a mechanical stage, a goniometer, apiezo component of the stage, an illumination of an electron beam, aprojection of the electron beam, an electromagnetic deflection of theelectron beam, and a movement of the electron beam.
 6. The controlsystem of claim 1, wherein the control system is further configured toregister the movement at a micron scale, a nanometer scale, or an atomicscale.
 7. The control system of claim 1, wherein the control system isfurther configured to simultaneously register movement associated with aplurality of regions of interest located in the sample underobservation.
 8. The control system of claim 1, wherein the controlsystem is configured to register the movement by referencing a templateimage of the region of interest against a remainder of the active areaof the sample.
 9. The control system of claim 8, wherein the controlsystem is further configured to manipulate a template image of theregion of interest over a predetermined period of time to generate acurrent morphology profile or a current intensity profile.
 10. Thecontrol system of claim 1, wherein the control system is furtherconfigured to capture the registered movement as a drift vectorassociated with one or more of: a structure of interest, a region ofinterest, and a background region, of the sample under observation.